picoCTF 2018 Write-up : Mr.Robot

Question: Mr.Robot

Information given:

Do you see the same things I see? The glimpses of the flag hidden away?

http://2018shell2.picoctf.com:15298

Hint

What part of the website could tell you where the creator doesn't want you to look?

Approach and Information required to solve this problem

When I first faced this problem, I tried to view the source of the page. However, there were not much of an information. I realized the problem is actually hard but then I realized the title of this question. Mr.Robot. Yes, "/robots.txt" came into my mind.

What is /robots.txt?

Website owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. You can think it's part of SEO, which stands for Search Engine Optimization. This tiny txt file is part of every website on the internet as well.

If you know more about what is /robots.txt or The Robots Exclusion Protocol visit following link, it explains in detail and how to use them. Robotstxt

Let's start Cracking:

When you follow the link, it guides you to the page which it looks like below

There are not much to see, go let's change the URL from

http://2018shell2.picoctf.com:15298

to

http://2018shell2.picoctf.com:15298/robots.txt

When you go to changed URL it will show you something similar to the image below

Disallow is used when the owner of the site doesn't want a robot to crawl.

what we need to do is that change /robots.txt to /c4075.html which will be like 

http://2018shell2.picoctf.com:15298/c4075.html

than it will guide to the following page which it looks like down below

That's the answer :D