Filter the data !!!

Fake Trum

@Alan-Kilborn Oh. I really have no experience in this matter. Because I thought, if there is any simple formula to shorten my time based on it. This question is really hard for me. I’m just worried you guys think that I am abusing it to make it difficult for you. I totally did not mean that.
Can you post my answer to this problem? Thanks

Alan Kilborn

@Fake-Trum said in Filter the data !!!:

Oh. I really have no experience in this matter.

So that’s a cop out. :-)
What if you have to do something similar next week?

We’ll get you started:

So, you could match the paired data items you are interested in with:

([0-9a-f]{32})</div><div class="ellipsis">(\w+)

Fake Trum

@Alan-Kilborn I really don’t understand what you’re talking about. But I am very grateful that you have been interested and answered my questions early.
I have tried to apply your command but this did not help me. I also do not know how to manipulate. I don’t really know much about Notepad ++ commands
How to upload images? I want to take a screenshot of my screen to show you my problem

Fake Trum

@Alan-Kilborn !

Alan Kilborn

@Fake-Trum

It’s not a bad start. :-)
Of course, we haven’t gotten to “replace” yet.

Are you using different data than you first provided? If I switch over to the Mark tab and mark the matches, I get this, as expected:

Can you see how the red marking covers the 2 pieces of data in each line that you are interested in? Try to duplicate this result.

Fake Trum

@Alan-Kilborn If that is difficult, is there any way to get this result:
.

nFpRJC5166
e7219089
dfh333
12butterfli
uZ005287
thegr81
cvbnuiop82
540322
jo08jo02
cyl1008
6eu5v7sLwI
19860613
peduna5
19770531
Fripouille7
021440
Strife1!
7yjv5lzO7Y
EDGAR14 \
hZnCinANX

Hash code, I’ll probably copy it by hand. It must be like that. Because the script fails, I have no other way

Fake Trum

@Alan-Kilborn If that is the result I get it would be great. Because at that time I only used instead </div> <div class = “ellipsis”> = accents: then I had the result I wanted. However, after marking the data as your image has uploaded, how to copy it?

PeterJones

@Fake-Trum ,

First, you need to actually read what Alan is writing to you. He’s helping you through step-by-step, not just giving you the final solution, because he wants you to learn how to do this yourself.

Second, and this one is much more important to me, you better have a pretty good explanation why you are having us help you harvest passwords from a page of MD5-hash / password pairs, and why this isn’t helping you get a list of passwords that you can try to break into accounts with. If you don’t have a good explanation for that, I will downvote every one of your posts right quick, because we do not help people steal passwords or break into accounts on this site.

Fake Trum

@PeterJones Without hiding anyone, I am using a Hash decode script, but the script fails resulting in such results. I can’t check it manually, because the data I have is over 10,000 lines. If you have other methods, can you suggest to help me overcome this problem?
@Alan-Kilborn Sorry, because you gave the command so succinct, I do not know how to operate: Replace or Search. Resulting in incorrect results.

Fake Trum

@Alan-Kilborn
226/5000
I don’t know why I did the same thing you did, but it only highlighted one result. I am very thankful for your help. Alright, maybe it bothers you too much. I found another way. You do not help me anymore. Thanks.

Alan Kilborn

Yea, @PeterJones raises a good point.
We can’t know for sure that we aren’t aiding nefarious purposes.
However, the pieces are all there, if they can be put together.
I think it is best for me to bow out at this point.
Sorry.

astrosofista

Hi @Alan-Kilborn, @PeterJones, All

I was about to answer this question when I noticed where it had led. Therefore, as a matter of prudence, I will not post my solution. Instead, I’ll describe it, because while it’s not original, it’s different from Alan’s.

It is a destructive method based on a logical disjunction, so it is suggested, obviously, to process a copy of the document. The regex reads line by line from the beginning of the document and if the line responds to the desired pattern -let’s say it contains the word ellipsis-, then the regex processes it and loads the desired data into two different groups, and if the line is not compatible then the regex ignores it, meaning that it will not be included into the replacement expression.

Consequently, after making a Replace All -just one mouse click-, only the 20 lines indicated by OP will remain in the document.

Have fun!

guy038

Hi @fake-trum, @alan-kilborn, @peterjones, @astrosofista and All,

Here is my attempt ! So, starting with the 142 lines of the initial HTML OP’s code, below :

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <title>MD5 Database - Nitrxgen</title>
  <meta http-equiv="content-type" content="text/html;charset=utf-8"/>
  <meta http-equiv="content-language" content="en-gb"/>
  <meta name="author" content="Nitrxgen"/>
  <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
  <meta name="description" content="A free, instant MD5 lookup service with over 1 Trillion password candidates."/>
  <link href="/css/general.css" rel="stylesheet" type="text/css"/>
  <link href="/css/md5db.css" rel="stylesheet" type="text/css"/>
  <script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.2/jquery.min.js" type="text/javascript"></script>
  <script src="/js/general.js" type="text/javascript"></script>
  <script src="https://www.google.com/recaptcha/api.js?render=6Lcd-pYUAAAAAAEb1ZAHtmdf3gJAmp5AQ8Pk28W9" type="text/javascript"></script>
  <script src="/js/md5db.js" type="text/javascript" defer></script>
</head>
<body>

<div id="s_main">
  <div id="s_head">
    <div></div>
    <a href="/">nitrxgen</a>
    <span style="background-color:rgba(255,255,255,.5);color:#FFF;height:35px;line-height:35px;"><b>STAY AT HOME</b></span>
  </div>
  <div id="s_pair">
    <div id="s_menu">
      <a href="/">Home</a>
      <a href="/contact/">Contact</a>
      <a href="/donations/"><img alt="" src="/img/star.png" style="vertical-align:-1px;"/> Donations</a>
      <hr/>
      <a href="/collatz/">Collatz Conjecture</a>
      <a href="/hashgen/">Hash Generator</a>
      <a href="/ntlmcase/">NTLM Case Corrector</a>
      <a href="/youtube_cc/">YouTube CC Downloader</a>
      <hr/>
      <a href="/factorialdb/">Factorial Calculator</a>
      <a class="current" href="/md5db/">MD5 Database</a>
    </div>
    <div id="s_body">

      <a class="section" name="md5_database">MD5 Database</a>
      <p>
        This is a look-up tool for typical unsalted <acronym title="Message Digest v5">MD5</acronym> cryptographic hashes. The
        database currently contains <acronym title="Or, exactly 1,127,962,538,784 passwords"><b>1.1+ trillion</b></acronym>
        passwords.
      </p>
      <p>
        To use this service, please use the <a href="https://www.nitrxgen.net/md5db_info/#api"><b>the dedicated API</b></a>.
      </p>

      <hr/>

      <p></p>
      <!-- <hr/> -->

      <a class="section" name="rfh">Recently Found Passwords</a>
      <p></p>
      <!-- IF YOU REALLY WANT TO CRAWL THIS BIT, GO FIND THE XML LOCATION IN THE JAVASCRIPT -->
      <!-- this bit MAY change in the future to use Server-Sent Events instead, so don't assume the XML will forever be available -->
      <div class="md5db_rfp">
        <div><div>Hash Value</div><div>Password</div><div>Hits</div><div>Found By</div><div>When</div></div>
        <div><div>37fdf1254303be28b01538692425c1a0</div><div class="ellipsis">nFpRJC5166</div><div>874</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>a28a5338b9bdf5946f164091b180d4c7</div><div class="ellipsis">e7219089</div><div>5</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>3b6b878850b5858771a83e0a270313f9</div><div class="ellipsis">dfh333</div><div>5</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>a93c6c7f2c3af560ed647a05a83318b8</div><div class="ellipsis">12butterflies</div><div>178</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>627e25817432ff801ccce621f39e4ff2</div><div class="ellipsis">uZ005287</div><div>6</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>43ba96c671cd4e4bec558fc82838dea9</div><div class="ellipsis">thegr81</div><div>177</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>0f5347444c2907992c7aea817b723644</div><div class="ellipsis">cvbnuiop82</div><div>183</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>428ca2a132023d13ec3d73af48ce2b6a</div><div class="ellipsis">540322</div><div>28</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>bddd7f92b46ae022c5b590a22459634d</div><div class="ellipsis">jo08jo02</div><div>101</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>c0a741e5e2fb2e3df81c1b003547825a</div><div class="ellipsis">cyl1008</div><div>9</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>45fd035ccac01f33baa48e91fb014dca</div><div class="ellipsis">6eu5v7sLwI</div><div>251</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>6f3a1642b22f6e816a2979963a3b2dff</div><div class="ellipsis">19860613</div><div>47</div><div>nitrxgen</div><div>0 secs ago</div></div>
        <div><div>d74864ba33eb47f5b9be5a6e37d9fc20</div><div class="ellipsis">peduna5</div><div>1</div><div>nitrxgen</div><div>1 sec ago</div></div>
        <div><div>2461b606819363e71e4f97b2b5ded126</div><div class="ellipsis">19770531</div><div>45</div><div>nitrxgen</div><div>1 sec ago</div></div>
        <div><div>4cd0efe4070757d2f6baeeac21cdb320</div><div class="ellipsis">Fripouille76</div><div>365</div><div>nitrxgen</div><div>1 sec ago</div></div>
        <div><div>112f30e72454a80f8a9e6168437cee4c</div><div class="ellipsis">021440</div><div>11</div><div>nitrxgen</div><div>1 sec ago</div></div>
        <div><div>c1775d5ee5751af2492bba1cc680fbd7</div><div class="ellipsis">Strife1!</div><div>234</div><div>nitrxgen</div><div>1 sec ago</div></div>
        <div><div>0912d4922fa5b8ec600b8ecaf3558293</div><div class="ellipsis">7yjv5lzO7Y</div><div>243</div><div>nitrxgen</div><div>1 sec ago</div></div>
        <div><div>d2f64eeb1a1ae8eeff8288e6ccc500d3</div><div class="ellipsis">EDGAR14$</div><div>105</div><div>nitrxgen</div><div>1 sec ago</div></div>
        <div><div>5adc95dfd45421d8e0522c90c54a2d4d</div><div class="ellipsis">hZnCinANX</div><div>118</div><div>nitrxgen</div><div>1 sec ago</div></div>
      </div>
      <p style="line-height: 20px;">&nbsp;</p>
      <hr/>


      <p>
        <b>GOOD NEWS</b> &mdash; A tool to allow users to paste hashes and have them checked against this database will be
        available very soon. It's 100% in the works. It will allow full speed lookups depending how many concurrent sessions
        there are. The time consuming part of this is making sure it won't be abused. Please check the
        <a href="/changelog/">Changelog</a> for further updates. &ndash; 25th November, 2019.
      </p>
      <hr/>
      <a class="section" name="statistics">Live Statistics</a>
      <p>
There is a grandtotal of <span id="stats_s1" style="font-weight:bold;">27,002,118,120</span> user hash requests made to this database, <span id="stats_s2" style="font-weight:bold;">178,851,726</span> are of unique hashes (about <span id="stats_s3" style="font-weight:bold;">0%</span> of grandtotal). Out of the grandtotal number of requests, <span id="stats_s4" style="font-weight:bold;">26,389,883,116</span> were successful or cracked (about <span id="stats_s5" style="font-weight:bold;">97%</span>). Regardingly only unique hashes, <span id="stats_s6" style="font-weight:bold;">143,451,392</span> were successful or cracked (about <span id="stats_s7" style="font-weight:bold;">80%</span>).      </p>

      <a class="section" name="gpu">GPU Processing</a>
      <p>
        Regular visitors may notice results showing in the table above as being found by "nitrx-gpu", these are cracked locally
        by GPU power in real time. When a hash you submit is not found, it will be queued for GPU cracking at some point in the
        future. Only when it is cracked by GPU will your unfound hash become found for the next time it's requested. The moment
        it gets cracked, it will appear in the table above. Similarly with passwords as "# NOT MD5 #" means the hash was cracked
        but not using the MD5 algorithm and will not be displayed.
      </p>

      <a class="section" name="information">Information</a>
      <p class="paper">
        <b>Main article</b>: <a href="/md5db_info/">MD5 Database - Information</a><br/>
        <b>Main article</b>: <a href="/md5db_info/#api">MD5 Database - API</a>
      </p>
      <p>
        The only data stored as a result of using this tool is the MD5 hash you willingly submit. Invalid form/API inputs are
        stored for the sake of monitoring unknown/malicious behaviour. Such things like IP addresses, cookies, HTTP headers,
        anything about you, your client or your connection, etc. are NOT stored.
      </p>
      <p>
        Do not contact me about hacking or accessing online accounts for any reason. Do not ask to access the list of passwords
        or hashes users submit. I do not condone any illegal or malicious activity; do not use this tool if that is your
        intention. Read more in the main article links above.
      </p>
      <p>
        This page loads an external script from Google called <i>reCAPTCHA v3</i> which is used to collect behavioural
        information of requests to determine if they're real users or bots. This information will eventually be used in new and
        upcoming features to combat automated requests from bots as it may place unwanted load on the server. For more
        information about Google's reCAPTCHA, please view Google's <a href="https://policies.google.com/privacy">Privacy Policy</a>
        and <a href="https://policies.google.com/terms">Terms of Service</a>.
      </p>

    </div>
  </div>

  <div id="s_tail">
    &copy; Copyright 2008-2020: Nitrxgen, all rights reserved.<br/>
    XHTML 1.0 valid and CSS3 valid.<br/>
    Source last modified 141 days ago.
  </div>
</div>

</body>
</html>

Then, @fake-trum, the following regex S/R :

SEARCH (?-is)^(?!.*[[:xdigit:]]{32}).*\R|^\h+<div><div>|(</div><div class="ellipsis">)|</div><div>.+

REPLACE ?1\:

with the Wrap around option ticked and the Regular expression search mode selected and a click on the Replace All button would give your expected text :

37fdf1254303be28b01538692425c1a0:nFpRJC5166
a28a5338b9bdf5946f164091b180d4c7:e7219089
3b6b878850b5858771a83e0a270313f9:dfh333
a93c6c7f2c3af560ed647a05a83318b8:12butterflies
627e25817432ff801ccce621f39e4ff2:uZ005287
43ba96c671cd4e4bec558fc82838dea9:thegr81
0f5347444c2907992c7aea817b723644:cvbnuiop82
428ca2a132023d13ec3d73af48ce2b6a:540322
bddd7f92b46ae022c5b590a22459634d:jo08jo02
c0a741e5e2fb2e3df81c1b003547825a:cyl1008
45fd035ccac01f33baa48e91fb014dca:6eu5v7sLwI
6f3a1642b22f6e816a2979963a3b2dff:19860613
d74864ba33eb47f5b9be5a6e37d9fc20:peduna5
2461b606819363e71e4f97b2b5ded126:19770531
4cd0efe4070757d2f6baeeac21cdb320:Fripouille76
112f30e72454a80f8a9e6168437cee4c:021440
c1775d5ee5751af2492bba1cc680fbd7:Strife1!
0912d4922fa5b8ec600b8ecaf3558293:7yjv5lzO7Y
d2f64eeb1a1ae8eeff8288e6ccc500d3:EDGAR14$
5adc95dfd45421d8e0522c90c54a2d4d:hZnCinANX

I suppose, @astrosofista, that is something similar to your regex S/R ;-))

Best Regards,

guy038

astrosofista

@guy038 said in Filter the data !!!:

I suppose, @astrosofista, that is something similar to your regex S/R ;-))

Hi @guy038, All:

I think so, as both approaches are destructive. Yours looks nicer, mine seems simpler in the sense that the techniques used are more basic -no look-arounds or POSIX character classes, for example- and also in terms of the logical structure, since the alternation has only two members, A|B. A describes the line to match, taking care to capture via negative classes both the hash and the password -so if it is only wanted the last one it is easy to deliver it-, and B deals with the unwanted lines, it’s a basic ^.*\R.

The replacement expression is, as you surely guessed, ?1$1\:$2\n.

Best Regards.

Alan Kilborn

@astrosofista @guy038

I think something is being missed here. First, should we truly be helping out the OP when we suspect we might only be aiding evil purposes? This is directed more to @guy038 because @astrosofista already acknowledged this.

Second, before it dawned on me (by @PeterJones hitting me over the head with it) that we might have a bad situation brewing, I already gave the answer for anyone that cared to follow it:

reference the other thread I linked early on, where @guy038 provided the general solution
use the regex I linked earlier in this thread which even included the capturing groups needed for the eventual (specific) solution!

guy038

Hello, @fake-trum, @alan-kilborn, @peterjones, @astrosofista and All,

Of course, I gave a solution, but you must admit that my post was quite succinct. As we say in France: the minimum trade union discourse ;-))

I mean that I wanted to express my disapproval and say that @fake-trum should have been more patient to fully examine our solutions, before giving up !

Perhaps it would have been better not to provide a solution at all, given that the PO did not want to get involved any further !

But the power and compactness of the regular expression code prevented me from doing so ;-))) So beautiful !

Cheers,

guy038

PeterJones

@Alan-Kilborn said in Filter the data !!!:

hitting me over the head with it

Well, I wasn’t trying to be violent to the regulars. I just saw the signs of hash/password pairs, and I couldn’t tell from the downloaded source code whether it was one of the “has my password been hacked” white-hat sites, or “here’s a list of password hashes for infiltrating poorly-written logins” black-hat-sites. The OP’s response wasn’t overly clarifiying.

Unfortunately, I realized last night while trying to fall asleep what regex I should have responded with, rather than my openly-antagonistic lingual response. It wouldn’t have been a solution to the OP’s question, but it might have helped the OP. See if you can figure out what it does before running it on the example data.

FIND: (?s)(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(.)?(\Z)?
REPLACE: (?1\x{49})(?2\x{20})(?3\x{57})(?4\x{49})(?5\x{4C})(?6\x{4C})(?7\x{20})(?8\x{4E})(?9\x{4F})(?10\x{54})(?11\x{20})(?12\x{42})(?13\x{52})(?14\x{55})(?15\x{54})(?16\x{45})(?17\x{20})(?18\x{46})(?19\x{4F})(?20\x{52})(?21\x{43})(?22\x{45})(?23\x{20})(?24\x{50})(?25\x{41})(?26\x{53})(?27\x{53})(?28\x{57})(?29\x{4F})(?30\x{52})(?31\x{44})(?32\x{53}\x{0D}\x{0A})(?33\x{0D}\x{0A}\x{0D}\x{0A}\x{2D}\x{2D}\x{20}\x{73}\x{69}\x{67}\x{6E}\x{65}\x{64}\x{2C}\x{20}\x{74}\x{68}\x{65}\x{20}\x{65}\x{78}\x{2D}\x{73}\x{63}\x{72}\x{69}\x{70}\x{74}\x{2D}\x{6B}\x{69}\x{64}\x{64}\x{69}\x{65})

Alan Kilborn

@PeterJones said in Filter the data !!!:

hitting me over the head with it

Slight language misinterpretation: I meant it more as me “getting hit by a lightening bolt of realization”…after you made it plain what could be going on.

astrosofista

Hi @Alan-Kilborn, All

What can I say! @guy038’s solution is very interesting and I don’t blame him for posting it. It’s worthy of analysis - at least I learned something - and I think it’s more geared towards the regulars on the list than to OP. As for my answer, it only describes a solution and is understandable only for those who know regular expressions, so I don’t see anything wrong with it.

To be honest, my first reaction was to suggest the direct selection of the passwords by means of Eko’s PS, something that doesn’t take more than a couple of seconds. However, as OP didn’t seem willing to learn anything and it was necessary to guide her/him to install the plugin and the script, I gave up this approach. But in the meantime I had noticed a different solution than yours, I tried it, it worked correctly, but didn’t publish it for the reasons seen.

Well, enough of this for me.

Now I would like to change the subject of the conversation a little, taking advantage of the fact that there are almost no new posts.

I have noticed that often the length of the regular expressions we are using exceeds by far the extension of the search and replacement fields, making it impossible to display the full expression. This limitation makes it difficult to analyze and understand other people’s expressions and to correct one’s own.

Taking into account that there is still some blank space in the Find window, wouldn’t it be a good idea to implement a line wrapping in the search and replacement fields, so that an expression exceeding 36 characters - the maximum displayable in my configuration - continues on the next line and so on until the expression is complete? Even if a limit is set for each field, say 3 lines, these would still give a better picture of the expression than one limited to a single line.

A bonus to facilitate the analysis and construction of regular expressions would be the implementation of a colored syntax to highligth groups and alternations at a glance - by the way, maybe I am not aware and this is currently feasible, you tell me.

Of course, I am aware that it is uncommon for the average user to run searches that go beyond the current length of the find field, let alone use regular expressions, so these features would not directly benefit most users. However I still find them valuables and I think they would be useful additions to Notepad++. Having made this caveat, I would like to hear your opinions. If these topics have been discussed before, I would appreciate links to those discussions.

Sorry for the long post :)

guy038

Hi, @astrosofista and All,

Personally, after dragging on the right, with the mouse, the Find dialog to its maximum, I’m able to type in up to 100 characters, with the monospaced search font ;-))

Best Regards,

guy038