Easily import Tweets into Drupal using JSON and Feeds

There are already many Drupal modules to display or import Tweets on your website, but most of them come with limitations: they just display the tweets instead of importing them. This makes it hard to filter, especially when you display a specific hashtag. In some cases, it’s pretty handy if you can quickly unpublish an unwanted tweet. The modules that do import tweets have other limitations. Or they create a custom table that makes it hard to display the tweets using Views. Or they only can pull 1 tag or user, instead of a combination of tags.

Let’s describe a current use case: we needed to display tweets combined with news items and blog posts in one overview, ordered by date. So we needed a way to actually store the tweets so we can filter and sort them. After trying some modules we switched to using Feeds. Using Feeds, we could harvest the clients’ Twitter RSS feed (example) and map the title, body and date field on a custom node type ‘Tweet’. This worked fine, but it had some limitations: only a few fields of the actual tweet are part or the RSS feed. It’s not possible to get information about hashtags, links or media being used in the tweet.

The client came up with the wish to display the photo’s being used in the tweet (Twitpic, yfrog, etc) instead of just showing the t.co url. To get this working, we switched to using the JSON feed (example) instead of the RSS feed. With the JSON feed, you can add a parameter to the URL (include_entities=true) to include all this information (more info). The feed also shows to which tweet this is a reply to, the user information, etc, etc. The problem was; how can we harvest JSON with Feeds?

With the help of the magnificent Feeds JSONPath Parser module it actually turned out to be fairly easy! Here’s how:

Create a node type ‘Tweet’

And add fields you want to harvest. We added (apart from the body and title) a field to store the ID, the Links (multiple Links field) and the Hashtags (multiple Term reference field to a vocabulary). If you need more information, just add extra fields.

Create a Feed Importer

  1. Download and enable Feeds & Feeds UI, Feeds JSONPath Parser (including JSONPath php plugin which you have to place in the module directory).
  2. Then navigate to /admin/structure/feeds/create to add an importer. I named it Tweets JSON (tweets_json).
  3. On the next page, you can edit the Importer settings. On Basic settings, I changed Periodic Import to ‘as often as possible’ so the tweets will be fetched on every cron run. Leave Fetcher as is. On Parser select the JSONPath parser.
  4. Go to the Node processor settings where you can map the imported feeds to your Content Type. Leave the other settings as is.
  5. Then go to Mapping. Here you have to add a mapping for each field you want to import. I added the following mappings:

    If you want to map other fields as well (author, reply_to, etc) you have to add those fields first to your content type before you can map them here.
  6. After adding the mappings, click on JSONPath Parser Settings on the left to describe your mappings. Here’s the tricky part.

Add JSON mappings

I won’t go into details of how the JSON structure is set up, but basically all you need to know is the path to each tweet. My use case only needed to import the tweets of a given user account, but obviously you can use every Twitter API request to make the selection you need.

If you use the Devel module (which you should) then the following snippet will give you the information you need:

$tweets = drupal_http_request('https://api.twitter.com/1/statuses/user_timeline.json?user_id=13994362&include_entities=true');
dpm(drupal_json_decode($tweets->data));

Note: to find the Twitter ID for a username, you can use TwIDder.

Here’s an example of the result for the @drupalcon account (the above command):

For the mapping, we need do understand the ‘tree’ of the data and map this tree on the Mappings interface. As you can see, every tweet is placed directly in the root of the request, so we can simply add $.* in the Context field.

Then, for the other fields, I used the following mappings:

  • Title - text
  • Body - text
  • Guid - id
  • Field_tweet_urls:url - entities.urls.*.expanded_url
  • Field_tweet_hashtags - entities.hashtags.*.text
  • Created - created_at
  • Field_tweet_id - id

I used the asterix for the URL and Hastags mappings because they are multiple fields (arrays).

Add the Import URL

Now you have created your Feed Importer you only have to add the Import URL. To do so, navigate to /import/add and enter the URL of the feed you want to import.

That’s all! And if you want to import tweets from several users, you can just add another Feed URL using the same importer :)

UPDATE: I've added a feature that you can use as starting point. It contains the content type and the feed importer. 

Next post

Use SASS and Compass to streamline your CSS development

Read More »

Comments

There are no new nodes. - said Drupal

This is easily one of the best-written tutorials ever. Thanks for that, first off. Any idea of this has changed at all? I am not able to get it to import at all. It seems to connect okay, but it just returns "There are no new nodes." but I have no nodes to begin with. I have a suspician on a couple of levels:

1. http vs. https - I had to change my Twitter status url from https (as you have above) to http to avoid a cURL permissions error I was getting. Once I changed it to http, I started getting the "no new nodes" message instead. Do you have any certificate settings on your site that I'm overlooking in my Drupal 7 instance that would allow me to use https and maybe fix this issue?

2. context of $.* - This is the other point of departure I'm curious about. I was wondering if it possibly should be something else. I've tried $..* or $.[*], etc, but nothing seems to work. They all return "no new nodes". 

Any tips you can provide would be immensly helpful. If I can give more debugging notes to help let me know. I'll be sure to post anything I come up with to fix it here as well.

Thanks again for the great post.

We did actually get a

We did actually get a variation of it working. We ended up not using the entities, and as soon as we removed them from the set-up it just worked. We are using oembed to display our tweets with Twitter's embed code. The added bonus is that Twitter pictures show up without any real work on our part. It took a few extra hoops and a cron task but it works nicely.

I haven't had time to double-check my initial set-up and see if the entities were the issue or maybe it was something else on my dev server. If you have time to share it as a feature that would be great. I can test it on my dev and see if I can identify where I screwed up. 

Needless to say, we have a working solution so there is no urgency to this and if you don't have time to pull the feature together no problem. I know with my workload right now it is hard to remember to eat let alone reply to blog comments. 

Thanks for the quick helpful response.

Cannoz execute PHP :-(

Hi,

thank you very much for this great tutorial, it works real fine!

But, when I try to execute your PHP Code with the PHP Execute Block I get the following error:

Notice: Undefined property: stdClass::$data in eval() (line 2 of/home/xyz/sites/all/modules/devel/devel.module(1285) : eval()'d code).

Any idea what that could be?!

Regards,

Tobi

Want the Problem, ...

Hi, that was easy, ... ;-) 

$tweets = drupal_http_request('https://api.twitter.com/1/statuses/user_timeline.json?user_id=13994362&include_entities=true');

For some reason there was HTML in the drupal_http_request in your snippet here. Got it working now, thanks again for this great tutorial. Regards, Tobi

Ah, my fault

It seemed that I placed the input filters on my blog in the wrong order. Fixed it now. By the way: using the PHP filter module is not really safe. I'd suggest creating a simple module with a custom hook_block_info and hook_block_view() instead. Example: http://drupal.stackexchange.com/a/5587/3209

API 1.1

Hi,

do you have any idea how Twitter API 1.1 with OAuth Authentification can be used?!

Regards,

Tobi

Same problem here. With API 1

Same problem here. With API 1.1:

- I used Drupal Feeds Oauth  (it exposes a new fetcher in the feed importer);

- I created an APP on twitter and obtained Oauth information: consumer key, consumer secret, request token URL, access token URL and authorize URL;

- inserted these info in the "HTTP Oauth Fetcher" settings under the new feed importer I created.

Creating a "tweet feed" (I attached the importer to a content type), Drupal throws me this warning: 

"Could not find OAuth access tokens for site twitter. You should probably authenticate first to access protected information."

I click on "authenticate first" to grant my user access to twitter api, redirected to twitter, accept the APP, grab the PIN twitter creates for me.

But then I'm stuck. What should I do with this PIN in Drupal?

"Feeds Oauth" 1.0 ask for "PECL::oauth" installed, but my shared hosting do not have extension installed. 

suggestions?

You shoud configure a

You shoud configure a callback URL in the application settings on dev.twitter.com. You have created one in the Drupal site when setting op the OAuthHttpFetcher. If you have this conigured Twitter will redirect you to that URL instead of showing a PIN code after authorizing the app.

This is awesome!!

Just to give thanks about this great tutorial. Even I've applied to a Drupal 6 site, it works!! For Drupal 6 users must say you have to install content taxonomy module to have the term reference field for the hahstags field, in Drupal 6 it's named "Content Taxonomy Fields". Hope this helps.

No sure

I get a 404 as well on that path. Maybe you've protected your tweets but can see them because you're logged in? I really don't know. 

New API Version

Hi,

I think, the main problem is, Twitter updates it API to Version 1.1!

To get tweets via feeds you need to use a token via OAuth.

Regards,

Tobi

Thank you

Hi,

thank you fot pointing to this! Will try it, on the other hand, I just found out today, Infojunkie is working on Feeds OAuth again. I will try out both this week!

Regards,

Tobi

Changed context

To get it to work with the search URL you will have to change the context to

$.results.*

instead of

$.*

 

By the way: your site was down ;)

Yes, I have. Just install the

Yes, I have. Just install the feeds_oauth module and change the HTTPFetcher into a OAuthHTTPFetcher. Register your website as an application on dev.twitter.com and you will receive the required URL's you need to configure in the OAuthHTTPFetcher settings. There you also create a callback URL that you will have to configure in your application settings on dev.twitter.com.

Next go to yoursite.com/import and you will see a message that you will have to authenticate at twitter. Just click on it and log in to twitter in the login screen that is opened. If everything is ok, you are redirected to the callback URL you created on your site and now you are all set to import tweets again.

Issues Connecting to OAuth

Hey there,

I'm actually having issues getting the OAuth portion working.  I believe I have the OAuth fetcher configured properly, but when I go to import the tweets and "authenticate", I get the following error:

"We tried hard, but did not get a request/temp token from the server."

Any thoughts?  Thank you in advance!

Special characters

This manual works like a charm.

Only problem is import chats with special charaters (smiley,etc..)

SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xF0\x9F\x98\x83',...' for column 'message' at row 1SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xF0\x9F\x98\x83',...' for column 'message' at row 1

Is there a way to filter these out?

 

Thnx

 

Variations of this

Hi there

 

Great Tutorial.. Thanks for posting this... Im having a little difficulty configuring however to not use a twitter feed but a JSON feed i have created.

Hope you could anser a couple of question as im still a little confused after reviewing your feature.

1) Where is the content type that the feed is actually writing to defined?

2) My JSON structure is as below

{"All_Jobs":{"MP":{"Jobs":[{"Jobref":"","Jobtitle":"", "Location-office":"","Location":"","Region ":"","Backoffice-location":"","Backoffice-region":"","Function":"","Industry":"", "Jobtype":"","ClientLogoURL":"", "Salary":"","Featured":"","Updated":"","Intro":"", "JobDescription":"","TheSuccessfulCandidate":"","AboutOurClient":"","ConsultantName":"", "ConsultantPhoneNo" :"", "ConsultantEmail":"", "JobStatus":"1","import":"1"}, ]}}} I have assumed by context will be  $.All_jobs.MP.jobs.* I have mapped fields like below  field_job_location ( content type field ) - $.All_jobs.MP.jobs.*.Location-office Does the above look correct?  AND im still getting no joy.... I am assuming this is related to my mapping but as previously said am confused how the feed knows to write to the job content type?  

Almost perfect

  1. The content type, like Tweets, can be defined in the Feeds Importer settings.
  2. If the context is $.All_jobs.MP.jobs.*, you only need Location-office, you don't want to repeat the whole other part (that's what the Context is for).

You don't need code

Hi Rob,

the snippet can be placed anywhere, even in template.php. It's just to show the content of the JSON request. All of the above can be achieved using the interface.