Scraping Movies
-
Scraping movies from IMDB takes a lot of time to complete. From loading movie posters to pictures of each actor it takes a lonnnggg time to add a movie. (and the bandwidth of course).
1) Is there any other way to control this scraping? Users can be provided be options to select the level of scraping. eg) Minimal, Moderate, Detailed.
An actor can have 10 images in IMDB. But some users might want to import just one image instead of everything.
2) Can we choose scraper other than IMDB?
3) Can support be added for Movie Organiser (http://www.movieorganizer.com/). Using this software I have scraped around 50 movies within an hour. But the inbuilt IMDB scraping in craigworks takes half-an-hour to scrap a single movie. (At 50kbps). -
Currently there is no way to control the number of images imported or to use another site other then IMDB. You have a very slow connection and CW was not designed for low bandwidths. One solution is to batch the imports and load overnight? Not sure if that would work for you but you can give it a try. If you know the IMDB ttxxxx code of the movies you want to import you can call an internal api call to import the movie in one call. Here is an example of a html page that will batch import movies:
<html>
<head>
<title>Import Movie</title>
<script language="Javascript">
function importMovie(ttcode){
try {
var xhr = false;
// Mozilla/Safari
if (window.XMLHttpRequest) {
xhr = new XMLHttpRequest();
} // IE
else if (window.ActiveXObject) {
xhr = new ActiveXObject("Microsoft.XMLHTTP");
}
xhr.open('GET', "http://localhost/CraigWorks/api/import?imdb=" + ttcode, false);
xhr.send();
} catch (e) {
}
}
</script>
</head>
<body>
<script type="text/javascript">
importMovie("tt0118571");
importMovie("tt0114319");
importMovie("tt0106977");
</script>
</body>
</html>
Hope that helps. -
surprisingly the above 'example' script works, just not as detailed as doing it manually (didn't import actors, less pictures etc.)
-
Please use new import movie utility introduced in 0.1.8d.
-
i use the import utility and added about 300 movies for queue. but when it come across some movie, the import froze so the queue didn't move at all. i tried deleting it, unqueue, pausing queue and delete, restart cwservice32, and even restart my pc, it still fails. so the queue got stuck and i cant even remove it.
any idea what to do?
EDIT:
it works fine again. I edited the data.script file and remove the movie line. -
I made a few fixes to the import utility in build 1816 where the import would hang. Please try it and see if it fixes your issue.
-
yup. 1816 definitely fixed that. thanks!!
Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
