Bash Script to read file and curl on each line and output if string found

Question

Trying to build a shell script which determines if the site is using PulseVPN or not. The script takes a input file which contains a list of URLs, iterate each URL line by line, perform cURL operation and grep for specific string. And if string found, display a echo message "Website using PulseVPN"

Here is my code -

Keyword="dana-na" # keyword to find for PulseSecure SSL VPN
while read -r line; 
do
if curl -s -L "$line" | grep "$keyword" > /dev/null
then
    # if the keyword is in the conent
    echo " The website $line is using Pulse Secure VPN" 
else
    echo "Pulse Secure VPN Not Found on $line"
fi
done < urls.txt

Problem The code is working but The problem here is when i include only one URL in input file. It gives correct result. But if i include more than one URL in input file, then it gives incorrect results. Can anyone explains why is this happening. Is anything wrong in code ?

Your code can't work because grep uses $keyword variable while you define Keyword (case is different). In general your loop should work ok but it will print The website $line is using Pulse Secure VPN for all urls due to keyword is empty — Maxim Sagaydachny
– Maxim Sagaydachny, Commented Dec 18, 2019 at 20:37
Note that grep -q "$keyword" is more efficient than grep "$keyword" >/dev/null, as the former (but not the latter) can stop as soon as the first match is found, instead of reading to the end of the file and writing content to /dev/null. — Charles Duffy
– Charles Duffy, Commented Dec 18, 2019 at 20:48
BTW, it's often helpful to run bash -x yourscript to trace the individual commands that get run and compare them against what you expect. For example, if you see your line being displayed with $'\r' characters at the end, then you know there's a newline type problem; if you see '' used instead of your keyword, you know to look at that (and can figure out the Keyword/keyword name mismatch issue), etc. — Charles Duffy
– Charles Duffy, Commented Dec 18, 2019 at 20:49
I'm not sure if curl consumes any data from its stdin, but you might want to try curl <&- ... to make sure it's not reading data intended for read. — William Pursell
– William Pursell, Commented Dec 18, 2019 at 21:12

Peymon · Accepted Answer · 2019-12-18 20:40:58Z

-1

It might be skipping over the last line due to how read works. Try

while read -r line || [[ -n $line ]];

Look at this post for more information.

answered Dec 18, 2019 at 20:40

Peymon

411 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Charles Duffy Over a year ago

If there isn't enough information in the question to allow a firm answer as opposed to a speculative one, comment instead of answering (and perhaps vote-to-close as lacking sufficient information). Moreover, if this were truly the problem, we'd want to close the question as duplicate of the one you're linking, instead of adding an answer that just points to a preexisting knowledge base entry.

Peymon Over a year ago

Gotta love the unending spiral of losing points because I can't comment and losing points for answering.

Collectives™ on Stack Overflow

Bash Script to read file and curl on each line and output if string found

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related