Facebook Posts Scraper (Tool) (It is up again!)

I am sure many of you would like some help scraping – scraping is a technique of extracting information from websites – the posts of a specific Facebook group. For example, when I was working on one of my early projects entitled Youth, ICTs, and Democracy in Egypt with the Technology & Social Change Group (TASCHA) at the UW – Information School, we needed to undergo a qualitative coding exercise for approximately 700 Facebook posts from the April Youth Movement Facebook group.  However, at the time of data collection, Facebook’s format did not enable users to browse through old posts. Additionally, the number of daily posts was immense; manual collection would have been prohibitively time-consuming. Therefore, I quickly realized the need for an application to save me  time while collecting Facebook posts. In order to collect Facebook posts, we developed an application using the Facebook Graph application programming interface (API), which is a way for developers to access Facebook data and build applications. 

This is the link to the application http://groupbrowser.azurewebsites.net/

How to Use the Application: 

1. At the beginning Log in with your Facebook account.

Screen Shot 2014-01-20 at 12.01.48 PM

2. After logging in, add the name of the Facebook group that you want to extract the posts from  ( I recommend copying it from Facebook)
3. Add the start Date of the posts you want to display
4. Add the end Date of the posts you want to display
5. Add the Number of posts
6. Click Submit

Screen Shot 2014-01-20 at 12.06.16 PM

The results are going to show in a bulleted list for readability and ease of use.

I really hope you could benefit from using this free application and feel free to ping me if you had any questions or concerns. Also, I would like to hear from you, what do you think of the App ? Would it be beneficial for you ?

Update: I just published a new post for the Facebook Page Scrapper!

Advertisements

56 thoughts on “Facebook Posts Scraper (Tool) (It is up again!)

  1. This is a fine scraper tool, but it only scrapes the last 250 posts. It won’t scrape the whole group if there are more postings.

      1. You can’t get it to scrap the entire group unless you manually by choosing multiple date ranges and specify the number of posts you need. Is it a very active group?

        Thanks,

    1. Hello Sang,

      Thank you for trying the tool. It worked with me when I used it a few months ago. I was using it to collect facebook posts from 2011. I will test it again and get back to you.

    1. Hello,

      unfortunately, this was made for groups only. but feel free to tweak the code to cover pages it shouldn’t be very hard. If you are unable to do so please contact me again and I will see what I can do.

      Thanks,

      1. I am not sure I will have time soon to update the code to include pages. I will keep you posted if I follow up with an update anytime soon.

    1. I am sorry about that. I will fix the application and share an update ASAP.

      Thank you for your interest in the application.

    1. I am sorry about that. I will fix the application and share an update ASAP.

      Thank you for your interest in the application.

    1. Hello,

      You need the name of the person who posted to be included with the posts? I will see what I can do.

      Thank you for your interest!

      ~Norah

  2. Thank-you, Norah. Your tool was quite helpful. FYI to others, you may need to play around with the dates to get all the posts. That’s a limitation of the underlying FB graph explorer and not Norah’s code. For example, I broke down two years worth of posts by choosing three different overlapping date ranges.

    Norah, it would be nice to at some point have the ID of the person doing the posts in order to keep track of the conversation. I appreciate your making this tool available to the public!

    1. Thank you for your comments and feedback ! I will look into adding the ID sometime this weekend.. I agree with you I think it would be really helpful.

  3. Norah, thank you very much for providing this application. I tried it, and it successfully scraped each unique post in my date range. However, it did not scrape the replies (or comments) to each post. Is there a way to scrape all the replies to each post? Thank you, again!

    1. Hi Sunnamarie,

      Thank you for your interest in the application and I am really glad it worked with you. I have been trying to updated the app to scrap facebook “Pages”. Once I am done with that I will add the replies and posts. It might take a while (because of my schedule) but I will send you an update once I am done.

      Thanks,
      norah

  4. Thanks for an amazing web based app Norah.

    I have few concerns though. Can it scrape all posts from a group altogether? Like from 1st post till the last.
    Its kinda annoying to change date ranges again and again and I’m getting around 240 posts only within any date range.

    Keep up with the amazing work!

    1. Thank you, Bilal for your comment!

      The main goal of developing this app was to scrape posts related to specific events, which is the reason for using the date range. I am not sure I will change it scrape the posts of the whole group because some groups get so many entries on daily bases that it might be hard to cramp it all in one process ( I want to stay mindful of people with slower internet connections).One possible solution, change the number of posts you want retrieved from the group ( I just tried that on a very active group) and it only retrieved 200 posts, which means it is not retrieving 1000. But it might works for you.) Please keep me posted and I will update you if I made any new updates.

  5. Thanks Norah for this app but I would prefer it to be used for pages. Hope you can inform me when you have updated it. Good luck

    1. Thank you for your interest. we’ve been working on the page collector but still no luck in making it collect the correct number of posts. I will keep you posted.

      Cheers,
      Norah

  6. Norah, that’s a great tool. But I want to scrap the complete post instead of just the title. Can you add that feature in the source code? If not possible, can you make the source code public so that I can tweak it myself.

  7. Hello Norah,

    I’ve noticed your application and one of NodeXL plugin scraps around the same parameters. Both your application and NodeXL from what I see on comments, when using date range, seems to not want to go beyond 200 posts (according to what I see on comments). Do you have any idea why this happens? I am also trying to scrape from 2/22/2014 to this date a facebook group but I’ve ben unsucessful. I’ve broke the year to months, and finally weeks. But then I realized even a 2 day went beyond 200.

    I am mostly interested on the relation to post and comment of one community. Any thoughts on that? Thank you.

    1. Hi Carlos,

      I am not sure why this is happening. When I was using this app in an earlier study it was working just fine. I will look into it in soon and get back to you.

      Thanks,

  8. Wow exactly what I’ve been looking for! Definitely beneficial to have this software! Thank you so much! I noticed it only worked for public groups, secret groups weren’t extracting for me (or I left a large gap between the dates to extract posts?).
    Anyway, here’s what would be cool to see in this for the future (and I’d pay good money for it!):
    Numbers of Likes per comment, threaded comments, ranked posts by likes while maintaining threads.
    I know these might seem complex to do but I’d pay money to have this work with those features!
    Despite those, really great work Norah 🙂 Thanks again.

    1. Thank you for you comments! To answer your question, the tool only can extract posts from public groups and most recently public pages. The authentication for private groups is not yet available from Facebook.

      I hope I can implement the updates you requested in the near future. I will keep you posted!

      Norah

    1. Hi! Thank you for your feedback.

      It could be done.One of the reasons I didn’t add the feature was to keep people’s posts and comments anonymous. If in the future this is something you want please write to me and I could make a small change to include it.

      Thank you!

  9. Hi Norah,
    First of all, thank you for creating this useful application.
    It’s awesome.
    By the way, could you make the results more organized?
    Perhaps you can create a grouping.
    Think of it as facebook post,
    where there’s a thread starter (TS) followed by people commenting it.
    Currently, I’m a bit confused since I don’t know which post a comment is commenting, so I don’t know the topic a person is commenting about.
    It doesn’t have to be the same as facebook, as long as it’s well organized, it doesn’t matter.
    Also could you retrieve the image files too? Sometimes people upload an image in his/her comment right?

    PS : I tried to retrieve the posts using ajax, but only limited to 10 object, do you know why? Thanks in advance.

    1. Hi,

      Thank you for your comment. I am working on a better way to represent the results and their type. I will post an update soon.

      Regarding your last comment, could you please clarify ? you could send me an email with an example.

      Thanks again!

  10. Hi Norah,

    Thank you for your reply. I studied the facebook SDK for Javascript, the old source codes have been replaced with the new ones, and now I’m able to get a better results.
    I’ll be waiting for your update and I’m sure lots of people also waiting for it.

    Cheers!

    1. Hi kukula,

      Thank you for your interest in using the tool. We just tried and the tool works if you specify “page” from the dropdown menu, copy paste the group name exactly and specify a date range where there actual posts.

  11. hi,
    Can i get the post from a Facebook user profile using above tool, if not then please tell me if there is some other way to do that.
    Thanks in advance.

    1. Hi,

      This is only for Facebook Pages and Groups. I am not sure I know of a similar app that works on Facebook users.

      Thank you!

    1. Hi Matteo, Thank you for your note. Could you please send the feedback through the feedback link. It helps with keeping track of suggestions.

    1. Hi,
      Thank you for your interest in my app. Yes, we designed it this way. It is not ethical to access closed groups information.

      Cheers,
      Norah

  12. Hi Nora, how can i do when the group that i would scrape has the same name of another group?
    For example: I’m interested for a group named “social media marketing”.
    Obviously isn’t the once.
    I tryed to insert url or id, but doesn’t work.
    How can I do?

    1. Hi,

      Thank you for your question. You can look at the URL the end of the URL is usually an Id or a title, copy that part and paste it to the text file. For example, the group ‘Internet Cat Research Group a/k/a Institute of Kitteh Studies’ the URL is ==> https://www.facebook.com/groups/1407788899455048/ If you took the last part ==> 1407788899455048 this is the unique group ID.

      Let me know if that works.

      Thanks,

    1. Hi Amanda, I will get back to you on this very soon. Might be able to do it in the coming days.

      Cheers,
      Norah

  13. Dear Norah Abokhodair,
    My name is Katrin Herms, I am a student in the field of cultural studies at the University of Saarbrücken/Germany and I am currently working on my Master degree’s thesis. My topic is Social Media Communication by right wing populists and I am supposed to undertake a content analysis of the posted entries + comments (including text, images, video and links to other articles, especially to other entries on Youtube and Twitter) of two Facebook pages with public access.
    Unfortunately, your scraping tool does not work. The program gives me the hint that I should reduce the number of entries. But even with 1 entry, I get this message. It would be very helpful to get your support by helping me to scrap the data so that I can undertake the content analysis of the following sites:

    Marine Le Pen (01.-31.12.2015)
    https://www.facebook.com/MarineLePen/?fref=ts

    PEGIDA (01.-31.01.2015 and 01.-30.09.2015)
    https://www.facebook.com/pegidaevdresden/?fref=ts

    Thank you a lot in advance!
    Kind regards, Katrin Herms

    1. Hi,

      Thank you for your interest! I will need to check the application authentication to know what is going on.

      Regarding your question on Twitter data, we developed a software to collect the Twitter data. Please refer to the paper I attached in the blog post to learn more about the collector.

      I will get in touch with an update once the tool is up and running again.

      Cheers,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s