Community
    • Login

    Type of duplicate lines

    Scheduled Pinned Locked Moved General Discussion
    25 Posts 5 Posters 4.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Sarah DuongS
      Sarah Duong @Terry R
      last edited by Sarah Duong

      @Terry-R If I use the alternative method as you said: All other lines with “:” will be removed. I meant to remove only lines that have only “:”. The lines that have the characters before and after the “:” are all kept. Do you understand me ???Take a closer look at my example, in my example there are only lines with “:”

      1 Reply Last reply Reply Quote 0
      • Terry RT
        Terry R
        last edited by

        @Sarah-Duong said in Type of duplicate lines:

        doesn’t help me eliminate duplicate lines

        I copied some of your examples and the built-in function “remove consecutive line” DID remove the duplicates, leaving just 1 of each group. If it’s not working for you then something else (NPP environment maybe, your examples aren’t the same as you have in the file, or the file type) is affecting the result.

        To post examples without the editor changing it use the information within the FAQ, specifically FAQ Desk: Request for Help without sufficient information to help you
        Often issues like you are having are found to be data related, such as information in the real data is not reflected in the examples. You will see some posts have their examples in a black window, the FAQ tells you how to do that.

        Terry

        Sarah DuongS 1 Reply Last reply Reply Quote 0
        • Sarah DuongS
          Sarah Duong @Terry R
          last edited by

          @Terry-R Many thanks for the support from you. However, as a newbie, I still have many shortcomings when posting questions. But what I posted on this topic is copied from my text file. I really want to find a way to fully upload it but I don’t know how to manipulate it. I have no idea what your answers are. Just because it doesn’t really help me. Hope you understand. Thanks, again

          1 Reply Last reply Reply Quote 0
          • Terry RT
            Terry R
            last edited by

            @Sarah-Duong said in Type of duplicate lines:

            what I posted on this topic is copied from my text file

            I understand that you may have copied the examples, however the editor you type in can interpret some characters and change the way examples show. See what I’ve done with your example below.

            4xtrader@tpg.com.au:
2emajnllc@gmail.com:
2emajnllc@gmail.com:
2emajnllc@gmail.com:
127victor@cox.net:
1talo@bluewin.ch:
10241024simon@gmail.com:
10241024simon@gmail.com:
4xtrader@tpg.com.au:
4xtrader@tpg.com.au:
10241024simon@gmail.com:
10241024simon@gmail.com:
4xtrader@tpg.com.au:

            I created this black window by using the ` character 3 times on a line before and a line after the examples. On an english keyboard it will generally be the key right before the number 1 key.

            We (all) on this forum are happy to help, however to do so you do need to be able to understand the questions we ask and be able to give good responses.

            Terry

            Sarah DuongS 2 Replies Last reply Reply Quote 0
            • Sarah DuongS
              Sarah Duong @Terry R
              last edited by Sarah Duong

              @Terry-R That is, if I want to post part of my text file here, I need to manipulate : Type ` character 3 times on a line before and a line after the examples? And will it be displayed on black window, same your ?

              1 Reply Last reply Reply Quote 0
              • Terry RT
                Terry R
                last edited by Terry R

                @Sarah-Duong said in Type of duplicate lines:

                ype 3 times the " "followed

                You will see that your post has been changed by the editor and the special character is missing. The character is like a backward facing quote character, often on the same key as ~ is.
                So 3 of this character on a line by themselves
                next line is the examples
                next line by themselves the same 3 characters.

                We don’t need all 20000 lines, but about 20-30 lines showing all 3 types of lines you want changed will be sufficient.

                Terry

                Sarah DuongS 1 Reply Last reply Reply Quote 0
                • Sarah DuongS
                  Sarah Duong @Terry R
                  last edited by

                  @Terry-R If I have multiple lines, I have to type multiple ampersands `? Or do I just need to type it at the beginning and at the end of the text file content?

                  1 Reply Last reply Reply Quote 0
                  • Terry RT
                    Terry R
                    last edited by

                    the 3 characters on a line. Then next line starts the examples for as many lines as necessary. the next line below the last line of characters is the 3 special characters again. So between the 2 lines of special characters are as many lines as you want.

                    Terry

                    1 Reply Last reply Reply Quote 0
                    • Sarah DuongS
                      Sarah Duong @Terry R
                      last edited by

                      @Terry-R and here my text :
                      1 / How to delete duplicate lines?
                      2/ Remove lines that have only “:”?
                      3/ How to choose the lines that have characters after “:”?

                      :
:
:
4xtrader@tpg.com.au:
2emajnllc@gmail.com:
127victor@cox.net:
1talo@bluewin.ch:
10241024simon@gmail.com:
4xtrader@tpg.com.au:
a.tworowski@o2.pl:sXOa61Dq
4xtrader@tpg.com.au:
1talo@bluewin.ch:
a.tworowski@o2.pl:sXOa61Dq
a_cameronsse@hotmail.com:jof6IutH
4xtrader@tpg.com.au:
a_nizam2032@yahoo.com:
aaaerealty@yahoo.com:
a.tworowski@o2.pl:sXOa61Dq
aaaerealty@yahoo.com:
10241024simon@gmail.com:
4xtrader@tpg.com.au:
10241024simon@gmail.com:
4xtrader@tpg.com.au:
:
4xtrader@tpg.com.au:
10241024simon@gmail.com:
a.tworowski@o2.pl:sXOa61Dq
4xtrader@tpg.com.au:
1talo@bluewin.ch:
2emajnllc@gmail.com:
4xtrader@tpg.com.au:
10241024simon@gmail.com:
4xtrader@tpg.com.au:
abradbery@gmail.com:q74Xpc0O
abrahamvthomas@hotmail.com:
a.tworowski@o2.pl:sXOa61Dq
:
aaron.r.cameron@gmail.com:
aaaerealty@yahoo.com:
aaron.r.cameron@gmail.com:
:
abradbery@gmail.com:q74Xpc0O
aaron.r.cameron@gmail.com:
abrahamvthomas@hotmail.com:
abradbery@gmail.com:q74Xpc0O
4xtrader@tpg.com.au:
a.tworowski@o2.pl:sXOa61Dq
abrahamvthomas@hotmail.com:
10241024simon@gmail.com:
1talo@bluewin.ch:
4xtrader@tpg.com.au:
10241024simon@gmail.com:
a-al-khaledi@hotmail.com:
abdullah.al.hajri0001@gmail.co:
a_cameronsse@hotmail.com:jof6IutH
abdullah.al.hajri0001@gmail.co:
abrahamvthomas@hotmail.com:
abdullah.al.hajri0001@gmail.co:
aaaerealty@yahoo.com:
abrahamvthomas@hotmail.com:
abdullah.al.hajri0001@gmail.co:
aaaerealty@yahoo.com:
abrahamvthomas@hotmail.com:
abrarahmed325@yahoo.com:
4xtrader@tpg.com.au:
abrarahmed325@yahoo.com:
4xtrader@tpg.com.au:
abrahamvthomas@hotmail.com:
4xtrader@tpg.com.au:
abrarahmed325@yahoo.com:
ac5.thomas@btinternet.com:
AccountingQB@brilloco.com:
abrahamvthomas@hotmail.com:
4xtrader@tpg.com.au:
abrahamvthomas@hotmail.com:
abrarahmed325@yahoo.com:
10241024simon@gmail.com:
abrahamvthomas@hotmail.com:
adelaideairportshuttles@gmail.:
adgrant6180@yahoo.com.au:
abrarahmed325@yahoo.com:
abrahamvthomas@hotmail.com:
ac5.thomas@btinternet.com:
AccountingQB@brilloco.com:
abrarahmed325@yahoo.com:
AccountingQB@brilloco.com:
abrahamvthomas@hotmail.com:
AccountingQB@brilloco.com:
abrarahmed325@yahoo.com:
AccountingQB@brilloco.com:
abrahamvthomas@hotmail.com:
a.tworowski@o2.pl:sXOa61Dq
abrarahmed325@yahoo.com:
aaaerealty@yahoo.com:
agarwalgaura@gmail.com:
abrahamvthomas@hotmail.com:
agarwalgaura@gmail.com:
agaskill@maalnet.com:
ageorgiev86@yandex.ru:dIYk0ONb
abrarahmed325@yahoo.com:
adgrant6180@yahoo.com.au:
adelaideairportshuttles@gmail.:
agarwalgaura@gmail.com:
abrarahmed325@yahoo.com:
agarwalgaura@gmail.com:
adgrant6180@yahoo.com.au:
abrahamvthomas@hotmail.com:
agilbert@hixworks.com:
adgrant6180@yahoo.com.au:
afoto@optonline.net:
agilbert@hixworks.com:
afoto@optonline.net:
agilbert@hixworks.com:
afoto@optonline.net:
adelaideairportshuttles@gmail.:
abrahamvthomas@hotmail.com:
agilbert@hixworks.com:
abrahamvthomas@hotmail.com:
adgrant6180@yahoo.com.au:
adelaideairportshuttles@gmail.:
advanced80@xtra.co.nz:
agilbert@hixworks.com:
agarwalgaura@gmail.com:
abrahamvthomas@hotmail.com:
abrarahmed325@yahoo.com:
agilbert@hixworks.com:
aipunts@yahoo.co.uk:pul8OBa4
agilbert@hixworks.com:
aaaerealty@yahoo.com:
agilbert@hixworks.com:
AccountingQB@brilloco.com:
agilbert@hixworks.com:
AccountingQB@brilloco.com:
aipunts@yahoo.co.uk:pul8OBa4
afoto@optonline.net:
agilbert@hixworks.com:
akisa5577@gmail.com:
AccountingQB@brilloco.com:
agilbert@hixworks.com:
aipunts@yahoo.co.uk:pul8OBa4
agilbert@hixworks.com:
abrarahmed325@yahoo.com:
agilbert@hixworks.com:
aj0312@my.bristol.ac.uk:
akisa5577@gmail.com:
agilbert@hixworks.com:
akisa5577@gmail.com:
aipunts@yahoo.co.uk:pul8OBa4
akisa5577@gmail.com:
agilbert@hixworks.com:
alagha.ahmad@gmail.com:
alain_delongchamp@yahoo.com:
agilbert@hixworks.com:
alain_delongchamp@yahoo.com:
agilbert@hixworks.com:
akisa5577@gmail.com:
alain_delongchamp@yahoo.com:
akisa5577@gmail.com:
alagha.ahmad@gmail.com:
AccountingQB@brilloco.com:
agilbert@hixworks.com:
akolanupaka@gmail.com:
alagha.ahmad@gmail.com:
agilbert@hixworks.com:
akolanupaka@gmail.com:
alamrozek@interia.eu:
alain_delongchamp@yahoo.com:
alamrozek@interia.eu:
alagha.ahmad@gmail.com:
alain_delongchamp@yahoo.com:
akisa5577@gmail.com:
alain_delongchamp@yahoo.com:
akisa5577@gmail.com:
alain_delongchamp@yahoo.com:
akisa5577@gmail.com:
akolanupaka@gmail.com:
alain_delongchamp@yahoo.com:
alamrozek@interia.eu:
alain_delongchamp@yahoo.com:
alan.james68@icloud.com:
Albert.Lau@eastwestbank.com:
alain_delongchamp@yahoo.com:
Albert.Lau@eastwestbank.com:
alain_delongchamp@yahoo.com:
abrarahmed325@yahoo.com:
alain_delongchamp@yahoo.com:
alan.james68@icloud.com:
alamrozek@interia.eu:
alan.james68@icloud.com:
ajurkovic@iinet.net.au:
Albert.Lau@eastwestbank.com:
alan.james68@icloud.com:
alamrozek@interia.eu:
ageorgiev86@yandex.ru:dIYk0ONb
alamrozek@interia.eu:
Alemannia@gmx.com:
alamrozek@interia.eu:
akolanupaka@gmail.com:
Alemannia@gmx.com:
alert@infoplasticsurgery.com:
alain_delongchamp@yahoo.com:
Albert.Lau@eastwestbank.com:
alain_delongchamp@yahoo.com:
Albert.Lau@eastwestbank.com:
albertrodriguez28@yahoo.com:
Alemannia@gmx.com:
alain_delongchamp@yahoo.com:
albertrodriguez28@yahoo.com:
aldis@hostnet.lv:
alan.james68@icloud.com:
alexrossouw196@gmail.com:
alan.james68@icloud.com:
alexrossouw196@gmail.com:
Alemannia@gmx.com:
alexrossouw196@gmail.com:
akisa5577@gmail.com:
Albert.Lau@eastwestbank.com:
aldis@hostnet.lv:
Albert.Lau@eastwestbank.com:
alexrossouw196@gmail.com:
aldis@hostnet.lv:
alain_delongchamp@yahoo.com:
alexrossouw196@gmail.com:
alert@infoplasticsurgery.com:
alexrossouw196@gmail.com:
Alemannia@gmx.com:
akisa5577@gmail.com:
Alemannia@gmx.com:
alexrossouw196@gmail.com:
alert@infoplasticsurgery.com:
akisa5577@gmail.com:
alert@infoplasticsurgery.com:
alektron@aol.com:
althielman@live.com:
altumbabicnahid@gmail.com:
albertrodriguez28@yahoo.com:
alexrossouw196@gmail.com:
albertrodriguez28@yahoo.com:
alexrossouw196@gmail.com:
alfred.kum@gmail.com:
alexrossouw196@gmail.com:
alfred.kum@gmail.com:
alert@infoplasticsurgery.com:
alfred.kum@gmail.com:
aman.di@hotmail.com:
amendol1@verizon.net:
alexrossouw196@gmail.com:
alistair@hexcollective.co.uk:
alfred.kum@gmail.com:
alistair@hexcollective.co.uk:
alfred.kum@gmail.com:
aman.di@hotmail.com:
abrarahmed325@yahoo.com:
aman.di@hotmail.com:
althielman@live.com:
AMERAHMED19@GMAIL.COM:
altumbabicnahid@gmail.com:
andreas.toerpel@web.de
alexrossouw196@gmail.com:
andreaszerbes@gmail.com:
ALJOAMAYA@GMAIL.COM:
alert@infoplasticsurgery.com:
aman.di@hotmail.com:
altumbabicnahid@gmail.com:
alexrossouw196@gmail.com:
andpanagiotop@gmail.com:
alfred.kum@gmail.com:
andpanagiotop@gmail.com:
alistair@hexcollective.co.uk:
alizenel@outlook.com:
aldis@hostnet.lv:
althielman@live.com:
alfred.kum@gmail.com:
ALJOAMAYA@GMAIL.COM:
alistair@hexcollective.co.uk:
aman.di@hotmail.com:
andpanagiotop@gmail.com:
aman.di@hotmail.com:
alan.james68@icloud.com:
andrewdonnellyjr@aol.com:qu48OcaN
andrzej.wencel@yahoo.com:
alfred.kum@gmail.com:
andrew.harnaga@hotmail.com:
andreas.toerpel@web.de:
alexrossouw196@gmail.com:
andrew.harnaga@hotmail.com:
andreaszerbes@gmail.com:
andrew@ezestream.com.au:
andrew.harnaga@hotmail.com:
altumbabicnahid@gmail.com:
andreas.toerpel@web.de:
andrewdonnellyjr@aol.com:qu48OcaN
andrzej.wencel@yahoo.com:
andrew.harnaga@hotmail.com:
anglinpaul@hotmail.com:
andrew.chaveriat@gmail.com:
alexrossouw196@gmail.com:
aman.di@hotmail.com:
andreas.toerpel@web.de:
antydoe@gmail.com:
anisessaid5@gmail.com:
andrew@ezestream.com.au:
andrew.harnaga@hotmail.com:
andrewdonnellyjr@aol.com:qu48OcaN
andreas.toerpel@web.de:
antydoe@gmail.com:
arash@42uag.com:
arolaxinvestor@gmail.com:
antydoe@gmail.com:
arolaxinvestor@gmail.com:
artallison@aol.com:
anisessaid5@gmail.com:
andreas.toerpel@web.de:
anisessaid5@gmail.com:
anglinpaul@hotmail.com:
andrew.harnaga@hotmail.com:
antuzla@outlook.com:
antydoe@gmail.com:
andpanagiotop@gmail.com:
ascrowe@wyoming.com:
arunasaste@gmail.com:
ash-1989-@hotmail.com:
andrzej.wencel@yahoo.com:
anglinpaul@hotmail.com:
ash-1989-@hotmail.com:
arash@42uag.com:
anuvu@ymail.com:
andrew.harnaga@hotmail.com:
antydoe@gmail.com:
artallison@aol.com:
andrew.harnaga@hotmail.com:
andrewdonnellyjr@aol.com:qu48OcaN
anglinpaul@hotmail.com:
ash-1989-@hotmail.com:
arunasaste@gmail.com:
argoman@hotmail.co.uk:
attention109@yahoo.com:
alexrossouw196@gmail.com:
antuzla@outlook.com:
attention109@yahoo.com:
andrzej.wencel@yahoo.com:
arunasaste@gmail.com:
arolaxinvestor@gmail.com:
antuzla@outlook.com:
asmoonlight@yandex.ru:
attention109@yahoo.com:
asmoonlight@yandex.ru:
ash-1989-@hotmail.com:
atinton@hotmail.com:
avysotsky@ukr.net:
arkadyokrezna@gmail.com:
axel@aadaum.de:
avysotsky@ukr.net:
arunasaste@gmail.com:
azyk1@yahoo.com:
ash-1989-@hotmail.com:
azyk1@yahoo.com:
anglinpaul@hotmail.com:
azyk1@yahoo.com:
b.costin23@gmail.com:
arunasaste@gmail.com:
ash-1989-@hotmail.com:
avysotsky@ukr.net:
attention109@yahoo.com:
avysotsky@ukr.net:
ash-1989-@hotmail.com:
attention109@yahoo.com:
avysotsky@ukr.net:
azyk1@yahoo.com:
Badykshanov@gmail.com:
b.costin23@gmail.com:
Badykshanov@gmail.com:
arunasaste@gmail.com:
avysotsky@ukr.net:
balsara@icloud.com:
banking5150@gmail.com:
antydoe@gmail.com:
alistair@hexcollective.co.uk:
avysotsky@ukr.net:
arunasaste@gmail.com:
ash-1989-@hotmail.com:
b.costin23@gmail.com:
ashley.brown@hushmail.com:
Badykshanov@gmail.com:
b.costin23@gmail.com:
avysotsky@ukr.net:
attention109@yahoo.com:
banking5150@gmail.com:
b.costin23@gmail.com:
axel@aadaum.de:
b.costin23@gmail.com:
banking5151@gmail.com:
azeezb22@gmail.com:
artallison@aol.com:
b.costin23@gmail.com:
b.rowsell@bell.net:
avysotsky@ukr.net:
banking5150@gmail.com:
Badykshanov@gmail.com:
banking5150@gmail.com:
avysotsky@ukr.net:
Badykshanov@gmail.com:
andrew@ezestream.com.au:
attention109@yahoo.com:
ash-1989-@hotmail.com:
baratina@gmx.net:
barnettos@yahoo.com:e38Ldp5C
bartekkuchnik@gmail.com:
baratina@gmx.net:
bartir@hotmail.com:
banking5150@gmail.com:
b.costin23@gmail.com:
banking5151@gmail.com:
banksdw@slu.edu:
azyk1@yahoo.com:
banking5150@gmail.com:
banking5151@gmail.com:
barakgr@live.com:
banksdw@slu.edu:
arunasaste@gmail.com:
b.rowsell@bell.net:
banking5151@gmail.com:
barakgr@live.com:
avysotsky@ukr.net:
banking5151@gmail.com:
barakgr@live.com:
Berganphoto@aol.com:
bertfrigo@gmail.com:
bengel1975@msn.com:
bertfrigo@gmail.com:
banksdw@slu.edu:
bartir@hotmail.com:
banking5151@gmail.com:
banking5150@gmail.com:
banksdw@slu.edu:
bimleshkumar@live.in:
bjh@yesyes.net:
bartir@hotmail.com:
banking5150@gmail.com:
bcteo@pegasus-it.com.sg:
BBJMcorp@aol.com:
banking5151@gmail.com:
BEDONEISM@HOTMAIL.COM:
bengel1975@msn.com:
BEDONEISM@HOTMAIL.COM:
beamugt@yahoo.com:
bddoliveiro@gmail.com:
beamugt@yahoo.com:
bartir@hotmail.com:
BEDONEISM@HOTMAIL.COM:
bjh@yesyes.net:
baratina@gmx.net:
blansford@lrshouston.com:fKBm16Pd
barakgr@live.com:
bcteo@pegasus-it.com.sg:
bjh@yesyes.net:
arunasaste@gmail.com:
bjh@yesyes.net:
blansford@LAMTexas.trade:
blansford@lrshouston.com:fKBm16Pd
bengel1975@msn.com:
blansford@lrshouston.com:fKBm16Pd
BEDONEISM@HOTMAIL.COM:
bobs114@yahoo.com.au:
bimleshkumar@live.in:
blansford@lrshouston.com:fKBm16Pd
bjh@yesyes.net:
barakgr@live.com:
bobs114@yahoo.com.au:
bertfrigo@gmail.com:
bengel1975@msn.com:
bobs114@yahoo.com.au:
blansford@lrshouston.com:fKBm16Pd
bobsoneau@yahoo.com.au:
bobwhite1946@yahoo.com:
barakgr@live.com:
blberger9@comcast.net:
blansford@lrshouston.com:fKBm16Pd
bohdarom@sbcglobal.net:
bobrabcd@frontier.com:
baratina@gmx.net:
bobsoneau@yahoo.com.au:
blansford@lrshouston.com:fKBm16Pd
bobsoneau@yahoo.com.au:
bertfrigo@gmail.com:
bigblckdg@aol.com:
bobwhite1946@yahoo.com:
bleda2_ju21@hotmail.com:
bohdarom@sbcglobal.net:
boonwee.hong@gmail.com:
boss_yuran@mail.ru:
bertfrigo@gmail.com:
boss_yuran@mail.ru:
billsilk@ozemail.com.au:
bobmedanovic@yahoo.com:
bobsoneau@yahoo.com.au:
bohetsj@gmail.com:
bobs114@yahoo.com.au:
banking5150@gmail.com:
bobs114@yahoo.com.au:
boonwee.hong@gmail.com:
bohdarom@sbcglobal.net:
boss_yuran@mail.ru:
boothmark71@hotmail.com:bFVi84Kx
boss_yuran@mail.ru:
bobs114@yahoo.com.au:
blansford@lrshouston.com:fKBm16Pd
boss_yuran@mail.ru:
botha.qatar@yahoo.com:
bobwhite1946@yahoo.com:
botha.qatar@yahoo.com:
blansford@lrshouston.com:fKBm16Pd
boothmark71@hotmail.com:bFVi84Kx
boss_yuran@mail.ru:
bobsoneau@yahoo.com.au:
boss_yuran@mail.ru:
bobwhite1946@yahoo.com:
boss_yuran@mail.ru:
botha.qatar@yahoo.com:
bowwybowwy@gmail.com:
brooksforex1529@yahoo.com:
bru.nico@alice.it:
boss_yuran@mail.ru:
bru.nico@alice.it:
brumbypat@hotmail.com:
bohdarom@sbcglobal.net:
bjh@yesyes.net:
boss_yuran@mail.ru:
camillopoland@gmail.com:
barakgr@live.com:
boss_yuran@mail.ru:
brchio@hotmail.com:
braykm01@yahoo.com:
bru.nico@alice.it:
brchio@hotmail.com:
brooksforex1529@yahoo.com:
bsrsolutions10@gmail.com:
carlo.paniccia@hotmail.com:
carlplunkett@hotmail.com:
bobwhite1946@yahoo.com:
brianchatting@yahoo.co.uk:
carlplunkett@hotmail.com:
brchio@hotmail.com:
botha.qatar@yahoo.com:
carlplunkett@hotmail.com:
boonwee.hong@gmail.com:
bowwybowwy@gmail.com:
boonwee.hong@gmail.com:
bobs114@yahoo.com.au:
cagoldman2005@yahoo.com:
boss_yuran@mail.ru:
beamugt@yahoo.com:
botha.qatar@yahoo.com:
carlplunkett@hotmail.com:
botha.qatar@yahoo.com:
cary.northup@gmail.com:
carlplunkett@hotmail.com:
cary.northup@gmail.com:
carlplunkett@hotmail.com:
carlcrabill@yahoo.com:
carlplunkett@hotmail.com:
camillopoland@gmail.com:
bowwybowwy@gmail.com:
carlplunkett@hotmail.com:
ccollins@semo.net:yd72XkjW
carlplunkett@hotmail.com:
booner2k@gmail.com:
casstlem@yahoo.com.au:
camillopoland@gmail.com:
cary.northup@gmail.com:
carlplunkett@hotmail.com:
cary.northup@gmail.com:
ccollins@semo.net:yd72XkjW
cary.northup@gmail.com:
ccollins@semo.net:yd72XkjW
cary.northup@gmail.com:
ccollins@semo.net:yd72XkjW
cdb07d@gmail.com:
botha.qatar@yahoo.com:
cbenjamin@cisolaw.com:
casstlem@yahoo.com.au:
botha.qatar@yahoo.com:
bobs114@yahoo.com.au:
bsrsolutions10@gmail.com:
ccollins@semo.net:yd72XkjW
bstarling@gmx.com:
cary.northup@gmail.com:
ccollins@semo.net:yd72XkjW
cary.northup@gmail.com:
ccollins@semo.net:yd72XkjW
casstlem@yahoo.com.au:
ccollins@semo.net:yd72XkjW
botha.qatar@yahoo.com:
cary.northup@gmail.com:
cdb07d@gmail.com:
bsrsolutions10@gmail.com:
cdb07d@gmail.com:
cdudek60@gmail.com:
ccollins@semo.net:yd72XkjW
cdudek60@gmail.com:
ccollins@semo.net:yd72XkjW
cdudek60@gmail.com:
cdb07d@gmail.com:
boss_yuran@mail.ru:
cdb07d@gmail.com:
cdudek60@gmail.com:
cdb07d@gmail.com:
cdudek60@gmail.com:
ccollins@semo.net:yd72XkjW
cemedia@aol.com:
cdudek60@gmail.com:
cdb07d@gmail.com:
cgsinvest@aol.com: 
huynhngoccuong@gmail.com:
idrzewicz@icloud.com:w0Re72Ht
info@simmtec.com:
ia_sho@abv.bg:
idrzewicz@icloud.com:w0Re72Ht
Hassamqazi7@gmail.com:
ihssass@hotmail.com:
idrzewicz@icloud.com:w0Re72Ht
haleelg@gmail.com:
gratica@att.net:gKb4EQp1
george@georgeharrison1.com:cgw3AMl8
hasco@personainternet.com:
idrzewicz@icloud.com:w0Re72Ht
Hassamqazi7@gmail.com:
ihssass@hotmail.com:
idrzewicz@icloud.com:w0Re72Ht```
                      1 Reply Last reply Reply Quote 0
                      • Sarah DuongS
                        Sarah Duong
                        last edited by

                        I used the line removal method as you instructed, it just shortened it. It seems that it only deletes duplicate lines close to each other. Conversely, if there is a line that is not on the side, it is not completely removed. It only helps to remove lines that have “:” unique. What I need is to delete the lines that have “:” and get the lines with all the characters before and after the “:” sign. I cannot use replacement in this case. Because it will replace all other lines. That I absolutely do not want.

                        1 Reply Last reply Reply Quote 0
                        • Terry RT
                          Terry R
                          last edited by

                          My suggestion for removing duplicate lines does NOT work as your data is different to the examples. In examples the duplicates are beside each other, your real data does not have that. If you were to sort your file lexicographically then my suggestion will work.
                          I’m not on my pc so cannot confirm the function but I think it’s in the same area as remove duplicate lines.

                          I’m sure we can have you sorted with the remaining problems very quickly. Can you answer, what do you want to do with the lines that have characters after the :, that was your #3 question.

                          Terry

                          Sarah DuongS 1 Reply Last reply Reply Quote 1
                          • Sarah DuongS
                            Sarah Duong @Terry R
                            last edited by Sarah Duong

                            @Terry-R In my text file, there are some lines that only have “:”. How to eliminate them? I cannot use alternative methods

                            ^: \ R

                            Because it replaces the “:” in other lines. You don’t seem to understand what I mean by this sentence.
                            In question 3, how to select only the lines with the characters after the “:” example.

                            a_cameronsse@hotmail.com: jof6IutH
abradbery@gmail.com: q74Xpc0O
a.tworowski@o2.pl: sXOa61Dq
ageorgiev86@yandex.ru: dIYk0ONb
aipunts@yahoo.co.uk: pul8OBa4

                            I don’t know how to make you understand what I mean. Sorry for my limited english

                            1 Reply Last reply Reply Quote 0
                            • guy038G
                              guy038
                              last edited by guy038

                              Hello, @sarah-duong, @Terry-r and All,

                              First of all, thanks for posting a real example of your text. However, I noticed that the very last line is :

                              idrzewicz@icloud.com:w0Re72Ht```

                              And I suppose that the correct line is rather :

                              idrzewicz@icloud.com:w0Re72Ht
                              • Regarding your question 2, to delete line with an unique colon char, the Terry’s regex S/R is NOT :
                              SEARCH   ^: \ R

REPLACE  Leave EMPTY

                              but, indeed :

                              SEARCH   ^:\R

REPLACE  Leave EMPTY

                              Assuming your example, after clicking on the Replace All button, 6 lines, with an unique :, are deleted !

                              • Regarding your question 1, to delete duplicates lines, you could use the following regex S/R :

                              SEARCH (?-s)^(.+\R)(?=(?s).+?^\1)

                              REPLACE Leave EMPTY

                              Again, after clicking on the Replace All button, from your initial text of 623 mail addresses, we get, at once, the expected text of 152 mail addresses, all different !

                              127victor@cox.net:
a_nizam2032@yahoo.com:
2emajnllc@gmail.com:
aaron.r.cameron@gmail.com:
abradbery@gmail.com:q74Xpc0O
1talo@bluewin.ch:
a-al-khaledi@hotmail.com:
a_cameronsse@hotmail.com:jof6IutH
abdullah.al.hajri0001@gmail.co:
4xtrader@tpg.com.au:
10241024simon@gmail.com:
ac5.thomas@btinternet.com:
a.tworowski@o2.pl:sXOa61Dq
agaskill@maalnet.com:
adgrant6180@yahoo.com.au:
adelaideairportshuttles@gmail.:
advanced80@xtra.co.nz:
agarwalgaura@gmail.com:
abrahamvthomas@hotmail.com:
aaaerealty@yahoo.com:
afoto@optonline.net:
aj0312@my.bristol.ac.uk:
aipunts@yahoo.co.uk:pul8OBa4
AccountingQB@brilloco.com:
agilbert@hixworks.com:
alagha.ahmad@gmail.com:
ajurkovic@iinet.net.au:
ageorgiev86@yandex.ru:dIYk0ONb
alamrozek@interia.eu:
akolanupaka@gmail.com:
Albert.Lau@eastwestbank.com:
alain_delongchamp@yahoo.com:
Alemannia@gmx.com:
akisa5577@gmail.com:
alektron@aol.com:
albertrodriguez28@yahoo.com:
amendol1@verizon.net:
abrarahmed325@yahoo.com:
AMERAHMED19@GMAIL.COM:
andreas.toerpel@web.de
alert@infoplasticsurgery.com:
alizenel@outlook.com:
aldis@hostnet.lv:
althielman@live.com:
ALJOAMAYA@GMAIL.COM:
alan.james68@icloud.com:
alfred.kum@gmail.com:
andreaszerbes@gmail.com:
altumbabicnahid@gmail.com:
andrew.chaveriat@gmail.com:
aman.di@hotmail.com:
andreas.toerpel@web.de:
anisessaid5@gmail.com:
andpanagiotop@gmail.com:
ascrowe@wyoming.com:
arash@42uag.com:
anuvu@ymail.com:
andrew.harnaga@hotmail.com:
andrewdonnellyjr@aol.com:qu48OcaN
argoman@hotmail.co.uk:
alexrossouw196@gmail.com:
andrzej.wencel@yahoo.com:
arolaxinvestor@gmail.com:
antuzla@outlook.com:
asmoonlight@yandex.ru:
atinton@hotmail.com:
arkadyokrezna@gmail.com:
anglinpaul@hotmail.com:
balsara@icloud.com:
antydoe@gmail.com:
alistair@hexcollective.co.uk:
ashley.brown@hushmail.com:
axel@aadaum.de:
azeezb22@gmail.com:
artallison@aol.com:
Badykshanov@gmail.com:
andrew@ezestream.com.au:
attention109@yahoo.com:
ash-1989-@hotmail.com:
barnettos@yahoo.com:e38Ldp5C
bartekkuchnik@gmail.com:
b.costin23@gmail.com:
azyk1@yahoo.com:
b.rowsell@bell.net:
avysotsky@ukr.net:
Berganphoto@aol.com:
banksdw@slu.edu:
BBJMcorp@aol.com:
banking5151@gmail.com:
bddoliveiro@gmail.com:
bartir@hotmail.com:
bcteo@pegasus-it.com.sg:
arunasaste@gmail.com:
blansford@LAMTexas.trade:
BEDONEISM@HOTMAIL.COM:
bimleshkumar@live.in:
bengel1975@msn.com:
blberger9@comcast.net:
bobrabcd@frontier.com:
baratina@gmx.net:
bigblckdg@aol.com:
bleda2_ju21@hotmail.com:
bertfrigo@gmail.com:
billsilk@ozemail.com.au:
bobmedanovic@yahoo.com:
bohetsj@gmail.com:
banking5150@gmail.com:
blansford@lrshouston.com:fKBm16Pd
boothmark71@hotmail.com:bFVi84Kx
bobsoneau@yahoo.com.au:
brumbypat@hotmail.com:
bohdarom@sbcglobal.net:
bjh@yesyes.net:
barakgr@live.com:
braykm01@yahoo.com:
bru.nico@alice.it:
brooksforex1529@yahoo.com:
carlo.paniccia@hotmail.com:
bobwhite1946@yahoo.com:
brianchatting@yahoo.co.uk:
brchio@hotmail.com:
boonwee.hong@gmail.com:
cagoldman2005@yahoo.com:
beamugt@yahoo.com:
carlcrabill@yahoo.com:
bowwybowwy@gmail.com:
booner2k@gmail.com:
camillopoland@gmail.com:
carlplunkett@hotmail.com:
cbenjamin@cisolaw.com:
bobs114@yahoo.com.au:
bstarling@gmx.com:
casstlem@yahoo.com.au:
botha.qatar@yahoo.com:
cary.northup@gmail.com:
bsrsolutions10@gmail.com:
boss_yuran@mail.ru:
ccollins@semo.net:yd72XkjW
cemedia@aol.com:
cdudek60@gmail.com:
cdb07d@gmail.com:
cgsinvest@aol.com: 
huynhngoccuong@gmail.com:
info@simmtec.com:
ia_sho@abv.bg:
haleelg@gmail.com:
gratica@att.net:gKb4EQp1
george@georgeharrison1.com:cgw3AMl8
hasco@personainternet.com:
Hassamqazi7@gmail.com:
ihssass@hotmail.com:
idrzewicz@icloud.com:w0Re72Ht
                              • Regarding your question 3, the Terry’s request seems justified :

                              Can you answer, what do you want to do with the lines that have characters after the :, that was your #3 question.

                              Indeed, you said :

                              3/ How to choose the lines that have characters after “:”?

                              But, once your lines are “chosen”, what next ?!

                              Now, it you want to easily point out these specific lines you could use the Mark feature :

                              • Click on the Search > Mark... menu option

                              • SEARCH (?-s):.+

                              • Tick the Bookmark line , Purge for earch search and Wrap around options

                              • Of course, select the Regular expression search mode

                              • Click on the Mark All button

                              => The lines, containing text after the : char, are bookmarked with a blue circle, and the text matched is highlighted in red !

                              • Then, some operations are possible on these bookmarked lines. Just select the sub-menu Search > Bookmark

                              For instance, using the Copy Bookmark Lines option, then a paste operation, here is the 14-lines list, from the modified text, without duplicate lines :

                              abradbery@gmail.com:q74Xpc0O
a_cameronsse@hotmail.com:jof6IutH
a.tworowski@o2.pl:sXOa61Dq
aipunts@yahoo.co.uk:pul8OBa4
ageorgiev86@yandex.ru:dIYk0ONb
andrewdonnellyjr@aol.com:qu48OcaN
barnettos@yahoo.com:e38Ldp5C
blansford@lrshouston.com:fKBm16Pd
boothmark71@hotmail.com:bFVi84Kx
ccollins@semo.net:yd72XkjW
cgsinvest@aol.com: 
gratica@att.net:gKb4EQp1
george@georgeharrison1.com:cgw3AMl8
idrzewicz@icloud.com:w0Re72Ht

                              Best Regards,

                              guy038

                              P.S. :

                              Once we are sure that your goals are achieved, we can give you some explanations on the regular expressions used ;-))

                              1 Reply Last reply Reply Quote 4
                              • Terry RT
                                Terry R
                                last edited by

                                @guy038 said in Type of duplicate lines:

                                Regarding your question 2, to delete line with an unique colon char, the Terry’s regex S/R is NOT :

                                @Sarah-Duong I think I see a problem you are having with the Regexes. Using Google translator is introducing spaces in any characters not determined to be words. Thus I used your original regexes in your first post and translated from English to Italian and this is what i got:
                                25fbef85-3da4-4ed5-b692-5db7e7472317-image.png

                                We can see that the original does NOT contain spaces, the translation does. So you need to be careful copying the regexes back to your language. by all means copy and translate so you can read our words to you. BUT, do not try to do the same for the regexes. Copy those and paste directly into NPP!

                                @guy038 I like your regex to remove duplicates, I had considered that but as I’ve said before I hate forward lookups due to the issue of it possibly failing completely. As a test I copied the examples (600+ lines) and made lots of copies in the same file. I got up to just shy of 200K lines and still the regex worked, I gave up trying to determine the limit at that point. Perhaps I’m being a bit harsh on that function!

                                Cheers
                                Terry

                                1 Reply Last reply Reply Quote 2
                                • guy038G
                                  guy038
                                  last edited by guy038

                                  Hello, @sarah-duong, @Terry-r and All,

                                  Terry, I would like to emphasize, in this post, the importance of choosing the right type of quantifier ( greedy, lazy or atomic ) in a regular expression !

                                  • From the initial text of @sarah-duong, above, which contains 623 lines, I duplicated it 325 times and I added a final line-break, at the very end of this test file. So, I obtained a file of 202,475 lines, for 5,016,375 bytes

                                  • Then, applying the regex S/R, with the lazy quantifier +? :

                                    • SEARCH (?-s)^(.+\R)(?=(?s).+?^\1)

                                    • REPLACE Leave EMPTY

                                  against this large text, I did get, after clicking on the Replace All button and 202,323 replacements ( In fact, suppressions ! ), in 3mn 57s,, on my old XP laptop, the very short expected text, of 3,711 bytes long, containing the 152 lines, all different !

                                  • Then, applying the regex S/R, where I changed the lazy quantifier +?, in the look-ahead, with the usual greedy quantifier + :

                                    • SEARCH (?-s)^(.+\R)(?=(?s).+^\1)

                                    • REPLACE Leave EMPTY

                                  against this same text, even after a 1-hour process about, no result occurred, although Notepad++ did not seem to get stuck !?

                                  So, I decided to run, again, this regex S/R, at 10h45 about, expecting a correct result, after some hours, when I was back home ! By chance, the process did stop, in the evening and has correctly deleted 202,323 lines, giving the expecting final file of 3,711 bytes long and 152 lines ;-))

                                  In order to know the exact time, used to execute these 202,323 replacements, by Notepad++, I simply opened the Process-Hacker v2.39.124 utility, double-clicked on the Notepad++ process to get its properties and, then, clicked, in the Threads tab, on the one with start address = notepad++0x12ab7b. See, below :

                                  a19e86ce-65cf-4fa9-ab2c-7c47da142597-image.png

                                  => The sum of the Kernel and User times, minus 2s about, due to N++ startup, indicates the time of the S/R : 6h 17m 52s !! Compare with the previous time of 3m 57s, as shown below ;-))

                                  129b0555-a3e7-45e9-869a-b881ece7fee7-image.png

                                  So, guys, could you repeat these two regex S/R, to know, even approximatively, with your configuration and OS, the time to process this UTF-8 test file, containing 325 times the initial text of @sarah-duong. So, a total of 202,475 lines and 5,016,375 bytes ! I quite curious of the results ;-))

                                  Best Regards,

                                  guy038

                                  1 Reply Last reply Reply Quote 3
                                  • Terry RT
                                    Terry R
                                    last edited by

                                    @guy038 said in Type of duplicate lines:

                                    So, guys, could you repeat these two regex S/R

                                    The PC config I tested on is Windows 10 64bit version 1607 (2016 LTSB) with a Intel i5-8600 and 8GB RAM.

                                    I used NPP 7.8.5 64bit.

                                    As requested I ran both your “lazy” and the “greedy” regexes. I had the exact same test file as you (same lines and byte size). The lazy S/R produced a time of 1m 23s. The greedy S/R produced the same result with a time of 1hr 54m (accurate to a minute only).

                                    Given your “old XP” system produced the 237s time for lazy and mine 83s, that’s a ratio of 0.35. If I do the same with the greedy S/R, given your time of 6h 17m 52s mine should be around 2hr 12m. My actual time of 1hr 54m is not too dissimiliar from that. So perhaps we can consider the speed of the regexes is “mostly” independent of OS version or even possibly CPU type, possibly little efficiencies in newer OS or CPU builds.

                                    I suppose the testing phase in the lookaheads is what takes the time. For the greedy regex, read all characters until end of file, test then drop 1 character, repeat until a solution found. Whereas the lazy regex just grabs 1 character then tests, continue until a solution found.

                                    As this test file has “at least” 325 copies of each and every “original” line the test is possibly not a very accurate one. Indeed the lookahead is “guaranteed” of finding a match no more than 623 lines ahead. In this situation the lookahead is never going to “fail” as we have seen in bigger files with a sparse number of duplicates.

                                    Terry

                                    1 Reply Last reply Reply Quote 3
                                    • Terry RT
                                      Terry R
                                      last edited by Terry R

                                      @Terry-R said in Type of duplicate lines:

                                      So perhaps we can consider the speed of the regexes is “mostly” independent of OS version or even possibly CPU type

                                      I should perhaps elaborate on this statement. I realise that the mere fact that my test results for only about 1/3rd that of @guy038 means there are efficiencies in OS and CPU, but I contend that they are mostly to do with GHz speed of the CPU, rather than microcode efficiencies.

                                      On “old XP” system would likely have a dual core (maybe a quad core) CPU with speed in the low GHz range. The i5-8600 has 6 cores with a speed of 3.1GHz.

                                      Could we suggest that the results have more to do with the number of cores and GHz speed, than efficiencies in microcode or the actual CPU design?

                                      Or is the question irrelevant as the answer to everything is “more speed/horsepower”!

                                      Terry

                                      Alan KilbornA 1 Reply Last reply Reply Quote 4
                                      • Alan KilbornA
                                        Alan Kilborn @Terry R
                                        last edited by

                                        @Terry-R said in Type of duplicate lines:

                                        …results have more to do with the number of cores…

                                        My 2 cents on this (and it could be worth much less) is that “throwing more hardware cores” at a problem does nothing for it unless software is written to utilize that hardware.

                                        Do we know that the regex engine used can “divide and conquer” this type of problem, by assigning pieces to multiple cores to work on, and then stitching together the results?

                                        …GHz speed…

                                        Well, yea, raw speed is going to have an impact, for sure.

                                        everything is “more speed/horsepower”

                                        Yes to the former, perhaps to the latter. :-)

                                        1 Reply Last reply Reply Quote 3
                                        • Sarah DuongS
                                          Sarah Duong
                                          last edited by Sarah Duong

                                          I want an example for you to understand:

                                          a_cameronsse@hotmail.com:jof6IutH
aipunts@yahoo.co.uk:pul8OBa4
:
:
abc:bcs:32da
:
a:
:a
orowski@o2.pl:sXOa6
:
onsse@hotmail.com:jof6
:
,:
a.tworowski@o2.pl:sXOa61Dq

                                          Now I want to get the results:

                                          a_cameronsse@hotmail.com: jof6IutH
aipunts@yahoo.co.uk: pul8OBa4
abc: bcs: 32da
a:
:a
orowski@o2.pl:sXOa6
onsse@hotmail.com:jof6
,:
a.tworowski@o2.pl:sXOa61Dq

                                          What do I have to do to get that result? I cannot use expressions instead:

                                           Find = : 
Replace with =

                                          That is the content of my question 3. Currently I have the answer: That is to use the expression:
                                          Find = ^ :
                                          Replace with = leave blank
                                          I have searched every corner of the forum to find this simple answer. Also because the language barrier makes it difficult for me
                                          I really want to convey my ideas to you. Unfortunately, Google translates incorrectly. Causing misunderstandings, or sometimes causing conflicts. Hope you understand me. Sincere thanks all for your interest in my topic

                                          Sarah DuongS 1 Reply Last reply Reply Quote 0
                                          • Sarah DuongS
                                            Sarah Duong @Sarah Duong
                                            last edited by Sarah Duong

                                            @Sarah-Duong

                                            Find = ^\:
Replace with : Leave blank

                                            I couldn’t type the punctuation so the sentence I wrote was missing.

                                            astrosofistaA 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            The Community of users of the Notepad++ text editor.
                                            Powered by NodeBB | Contributors