• Login
Community
  • Login

sort file removing duplicates possible?

Scheduled Pinned Locked Moved Help wanted · · · – – – · · ·
75 Posts 5 Posters 46.6k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    Scott Sumner @patrickdrd
    last edited by Oct 5, 2017, 2:58 PM

    @patrickdrd

    For the DBServerIP=10.1.249.215 case you mentioned, if you don’t do the sort first, then a .215 line is the last line of the data you turn over to the regular expression replace operation. Thus it lacks the trailing line-ending that the earlier occurrence of a similar line has. Same issue as the .185 case…

    1 Reply Last reply Reply Quote 0
    • P
      patrickdrd
      last edited by Oct 5, 2017, 4:04 PM

      thanks a ton!

      I’d a like a more “generic” approach, so I called trim 42056 to clear empty lines first,
      because sorting puts an empty line at the top if is finds one,
      then going on as you suggested

      S 1 Reply Last reply Oct 5, 2017, 5:09 PM Reply Quote 0
      • S
        Scott Sumner @patrickdrd
        last edited by Scott Sumner Oct 5, 2017, 5:10 PM Oct 5, 2017, 5:09 PM

        @patrickdrd

        I’m glad you have a solution. Yeah, the empty line thing with sorting is rather bad, but the end-user can’t control this behavior of the sorting. I think I’ve changed my mind and it is probably best to alter the regular expression a bit in order to handle the situation where there is a duplicate-but-without-line-ending at the end of the file. So I’d suggest changing it to:

        (?-s)^(.*)\R(?s)(?=.*^\1(?:\R|\z))

        I’ve done two things to this regex:

        • I removed the (?: and ) around the first \R (a simplification discussed earlier so no need to say any more here)
        • The final \R was changed to (?:\R|\z) (see discussion below)

        The \R|\z part is what allows an almost-duplicate at the end-of-file-without-line-ending to be detected. The new part to this is the \z which roughly means “match only at the very end of the data”.

        The (?: and ) was added so that the | only affects the \R that precedes it and the \z that follows it

        1 Reply Last reply Reply Quote 2
        • G
          guy038
          last edited by Oct 6, 2017, 7:32 PM

          Hello @patrickdrd, @scott-sumner and All,

          Generally speaking, when you want to remove duplicate lines, from a PREVIOUSLY SORTED list, just use this simple regex S/R, below :

          SEARCH (?-s)(.*\R)\1+

          REPLACE \1

          This regex is quite fast, because, in case of numerous duplicates, the part \1+ grabs all the duplicates ( with their EOL characters ), at once and just rewrites the first item of each block :-))

          IMPORTANT : the last item of your sorted list must be followed by EOL character(s) !

          Cheers,

          guy038

          1 Reply Last reply Reply Quote 3
          • P
            patrickdrd
            last edited by May 31, 2018, 10:40 AM

            after trying both the macro and the textfx solution for a long time,
            I’ve seen that still ultraedit’s sorting works much better than both of them,
            I first rejected the macro for textfx’s favor but I found out lately that neither the latter does a good job, at least I prefer the sorting done by ultraedit ,sorry,
            I don’t have an example at the moment, I’ll post again when I do

            C 2 Replies Last reply May 31, 2018, 12:48 PM Reply Quote 0
            • C
              Claudia Frank @patrickdrd
              last edited by May 31, 2018, 12:48 PM

              @patrickdrd

              sorry, did not read the whole thread but what about a python one liner?

              editor.setText('\r\n'.join(list(set(editor.getText().splitlines()))))
              

              Cheers
              Claudia

              1 Reply Last reply Reply Quote 0
              • C
                Claudia Frank @patrickdrd
                last edited by May 31, 2018, 12:52 PM

                @patrickdrd

                forgot sorting

                editor.setText('\r\n'.join(sorted(list(set(editor.getText().splitlines())))))
                

                Cheers
                Claudia

                1 Reply Last reply Reply Quote 1
                • P
                  patrickdrd
                  last edited by May 31, 2018, 1:05 PM

                  thanks, I’m keeping that too and I’ll let you know

                  1 Reply Last reply Reply Quote 0
                  • P
                    patrickdrd
                    last edited by patrickdrd May 31, 2018, 1:23 PM May 31, 2018, 1:22 PM

                    ok, I tested, this doesn’t work either somehow, at least doesn’t work like ue’s one, i.e.

                    I tested with easylist from: https://easylist.to/easylist/easylist.txt (69883 lines),
                    ue’s function cuts it to: 69238

                    npp results:
                    python scipt: 69818
                    macro (scott) is a bit slow and results in 19610 lines (!) and
                    textfx results in 69818 as well

                    guy038 regular expression results in 28109 lines,
                    according to my experience, I bet that ue’s result is the correct one,
                    at least to my taste

                    C 1 Reply Last reply May 31, 2018, 1:24 PM Reply Quote 0
                    • C
                      Claudia Frank @patrickdrd
                      last edited by May 31, 2018, 1:24 PM

                      @patrickdrd

                      could you, by any chance, upload the ue cutted list?
                      To see the differences.

                      Cheers
                      Claudia

                      1 Reply Last reply Reply Quote 0
                      • P
                        patrickdrd
                        last edited by May 31, 2018, 1:28 PM

                        yes, of course, please tell me where, pastebin doesn’t work, it’s blocked here (at work),
                        any other suggestions?

                        1 Reply Last reply Reply Quote 0
                        • C
                          Claudia Frank
                          last edited by Claudia Frank May 31, 2018, 1:37 PM May 31, 2018, 1:37 PM

                          actually pastebin is my first choice as well and haven’t used others for quite some time now.

                          Heard about

                          https://www.zippyshare.com/
                          https://www.sendspace.com/

                          should be good and anonymous but haven’t tried it so far.

                          Cheers
                          Claudia

                          1 Reply Last reply Reply Quote 0
                          • S
                            Scott Sumner
                            last edited by May 31, 2018, 1:38 PM

                            Yea, I used to be a fan of regular expression replacement when doing this, but with “larger” datasets there always seems like there is so much tweaking and experimentation needed to get it right (for a particular dataset) that it is hardly worth it, unless you like playing with regular expressions all day instead of solving a particular problem and moving on quickly.

                            A Pythonscript solution such as @Claudia-Frank 's seems fine…

                            1 Reply Last reply Reply Quote 1
                            • P
                              patrickdrd
                              last edited by May 31, 2018, 1:55 PM

                              ok, but why results are inconsistent (with large datasets)?

                              @Claudia-Frank, unfortunately I’m not able to access zippyshare either,
                              so I’ll upload to pastebin from home if we don’t find another solution

                              C S 2 Replies Last reply May 31, 2018, 2:22 PM Reply Quote 1
                              • C
                                Claudia Frank @patrickdrd
                                last edited by May 31, 2018, 2:22 PM

                                @patrickdrd

                                I installed the ue trial version but can’t find the menu item to delete the duplicates.
                                Is there anything I need to install in addition or am I blind and don’t see the obvious?

                                Cheers
                                Claudia

                                1 Reply Last reply Reply Quote 0
                                • C
                                  Claudia Frank
                                  last edited by May 31, 2018, 2:29 PM

                                  ok - found it - obviously blind :-D

                                  Cheers
                                  Claudia

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    Scott Sumner @patrickdrd
                                    last edited by May 31, 2018, 2:35 PM

                                    @patrickdrd said:

                                    ok, but why results are inconsistent (with large datasets)?

                                    Various reasons, sometimes a regular expression approach to this needs to be refined to match the data better before it works well. If you search up some other threads on this topic you can trace through the evolution of a regex approach on certain datasets. However, I suspect you just want to get a workable solution and move on…and I fully endorse that. I’m tired of trying to use regex for this kind of thing. :)

                                    BTW, @Claudia-Frank going the extra mile…installing UE trial version just to track this down…nice!

                                    C 1 Reply Last reply May 31, 2018, 3:00 PM Reply Quote 0
                                    • C
                                      Claudia Frank @Scott Sumner
                                      last edited by May 31, 2018, 3:00 PM

                                      @Scott-Sumner

                                      … boots are made for walking … :-)

                                      I’m confused about how UE does sort and delete duplicates.
                                      Used default settings.

                                      Only sorting

                                      Sorting and deleting duplicates

                                      There is obviously something wrong about UEs algorithm, isn’t it?
                                      And the version I used cut the list to 63732 lines.

                                      Cheers
                                      Claudia

                                      1 Reply Last reply Reply Quote 0
                                      • P
                                        patrickdrd
                                        last edited by May 31, 2018, 3:13 PM

                                        so… another number? amazing!

                                        I’ve got version 16 which is lite somehow

                                        1 Reply Last reply Reply Quote 0
                                        • C
                                          Claudia Frank
                                          last edited by May 31, 2018, 3:17 PM

                                          me too - that is actually the latest one available for ubuntu.

                                          Cheers
                                          Claudia

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post
                                          The Community of users of the Notepad++ text editor.
                                          Powered by NodeBB | Contributors