{"id":25,"date":"2021-04-29T21:46:22","date_gmt":"2021-04-29T20:46:22","guid":{"rendered":"https:\/\/ambitiousdeveloper.com\/?p=25"},"modified":"2021-04-29T21:46:22","modified_gmt":"2021-04-29T20:46:22","slug":"a-small-program-2","status":"publish","type":"post","link":"https:\/\/ambitiousdeveloper.com\/index.php\/2021\/04\/29\/a-small-program-2\/","title":{"rendered":"A small program"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">About 3 weeks ago I was invited in to a job as a creative python developer. I had applied for this job a while ago, and didn&#8217;t expect to get in since I have no real web devlopment experience with Python so I was pleasantly surprised to get an invitation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To this invitation I had to amongst others create a program that:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&#8220;Can download all links on a page, and save it in a file in a structured way. You must use python3 and assume you have all modules available for the task&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So I started down the road of finding out what modules\/libraries I could use for this task .I found something called BS4(BeautifulSoup)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"http:\/\/www.crummy.com\/software\/BeautifulSoup\/\">Beautiful Soup<\/a> is a Python library for pulling data out of HTML and XML files. It works with a parser to provide navigating, searching, and modifying the parse tree.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So I was already down the &#8220;lets create the program now!&#8221; when I stopped and thought. Hmm.. maybe I should just write down in rough steps what this program should do, right? So this is what I came up with:<\/p>\n\n\n<ul>\n<li>Create an object for the webpage<\/li>\n<li>Download the webpage from the object before in text format<\/li>\n<li>Create a soup with BS4<\/li>\n<li>Open a new csv file on disk and set it to write mode<\/li>\n<li>create a for loop that iterates through the site searching for links<\/li>\n<\/ul>\n\n\n<p class=\"wp-block-paragraph\">I fired up my editor and started with getting the dependencies, requests and bs4. With the dependencies done I setup some simple variables to store the webpage, the url request stored in text format, the soup object for our downloaded url. I then needed to create a local file in csv format, to store the data. Now the for loop took a little time. I needed to find out what I was looking for, so I did some online checking on what tags I could look for to find links on a page. The a tag followed by href gives me the links on the page. Now creating the for loop I look through our soup object for the a-tag, then save all the links found with the href attrib as data. Then I write the data to the file I opened before and comma seperate the links, and make line shifts as well to make it easy to read. Then for the fun of it I added a print statement to print out the links. Finally we close the file(always remember to close the file I am told..)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import requests\nimport bs4\n\nurl = \"&lt;webpage url&gt;\"\ndownload_url = requests.get(url).text\nsoup = BeautifulSoup(download_url,\"html5lib\")\n\n# Opening a file in write mode\nf = open(\"list-of-links.csv\", \"w\")\n\nfor links in soup.find_all('a'):\n    data = links.get('href')\n    f.write(data)\n    f.write(\",\")\n    f.write(\"\\n\")\n    print(links.get('href'))\n# Always remember to close it, when you are done\nf.close()<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">And so I was done with the first task.. I went to the interview and explained my thought process on the task, and answered other questions regarding what modules I used and why. It was fun even though it didn&#8217;t pan out.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In my next post I will show you some data science that I worked on a while back. Looking forward to it next week!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>About 3 weeks ago I was invited in to a job as a creative python developer. I had applied for this job a while ago, and didn&#8217;t expect to get in since I have no real web devlopment experience with Python so I was pleasantly surprised to get an invitation. To this invitation I had&hellip;<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[3,2],"class_list":["post-25","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-python","tag-webcrawler"],"_links":{"self":[{"href":"https:\/\/ambitiousdeveloper.com\/index.php\/wp-json\/wp\/v2\/posts\/25","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ambitiousdeveloper.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ambitiousdeveloper.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ambitiousdeveloper.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ambitiousdeveloper.com\/index.php\/wp-json\/wp\/v2\/comments?post=25"}],"version-history":[{"count":1,"href":"https:\/\/ambitiousdeveloper.com\/index.php\/wp-json\/wp\/v2\/posts\/25\/revisions"}],"predecessor-version":[{"id":26,"href":"https:\/\/ambitiousdeveloper.com\/index.php\/wp-json\/wp\/v2\/posts\/25\/revisions\/26"}],"wp:attachment":[{"href":"https:\/\/ambitiousdeveloper.com\/index.php\/wp-json\/wp\/v2\/media?parent=25"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ambitiousdeveloper.com\/index.php\/wp-json\/wp\/v2\/categories?post=25"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ambitiousdeveloper.com\/index.php\/wp-json\/wp\/v2\/tags?post=25"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}