Jump to content


Photo
- - - - -

making custom grabbers

grab grabber custom

  • Please log in to reply
4 replies to this topic

#1 brianatthebeach

brianatthebeach

    Member

  • Members
  • PipPipPip
  • 107 posts

Posted 20 August 2017 - 10:15 PM

I decided to try my hand at learning how the avs grabbers work and to make my own custom grabber.  Also learning about the grabbers could help me fix when one or another stops working.

 

So a user said he uses gayforit.eu and I decided to start there.

 

I noticed the grabbers get the video url by using

$content = $this->curl->saveToString($videopagewhereVURLis);

In various grabbers it looks like you get a page url from the target website and then use regular expressions to find the actual video url in the page.

 

So in general, it looks like if you can find a page on the site you want to grab from that gives up the vurl, then you use a regular expression to get vurl which is used by cURL to grab the video.

 

So now I looked on gayforit and using both the video page and embed player page, I wrote the page results of $content to a log file.  There is no indication of an actual video url.

 

I know they use nginx and I see a lot of javascript on the pages.  I am getting the impression that maybe they are grabbing the video url using javascript and never giving up the video url in the html.

 

You can see the actual NGINX video url using chrome developer tools and watching the network activity.

 

Am I missing something or is this type of site not grabber friendly?

 

Thanks,

Bran


Edited by brianatthebeach, 20 August 2017 - 10:18 PM.


#2 theone

theone

    Senior Member

  • Members
  • PipPipPipPip
  • 371 posts

Posted 21 August 2017 - 02:27 AM

You can get the vkey with curl and then you can extract the links using the following url

https://www.gayforit.eu/jwplayer6/jwplayer.normal.js?vkey=XXXX

Edited by theone, 21 August 2017 - 02:28 AM.


#3 theone

theone

    Senior Member

  • Members
  • PipPipPipPip
  • 371 posts

Posted 21 August 2017 - 02:37 AM

Though, from the way they are serving files you could also just write a very basic script to just insert the id from the url in order to fetch the file and not even use curl at all.

https://media-cdn.gayforit.eu/content/XXXXXXX/mp4/XXXXXXX_360p.mp4
https://media-cdn.gayforit.eu/content/XXXXXXX/mp4/XXXXXXX_480p.mp4
https://media-cdn.gayforit.eu/content/XXXXXXX/mp4/XXXXXXX_720p.mp4

Edited by theone, 21 August 2017 - 02:38 AM.


#4 brianatthebeach

brianatthebeach

    Member

  • Members
  • PipPipPip
  • 107 posts

Posted 21 August 2017 - 02:59 AM

I thought of constructing the the links based on the way gayforit was serving their vids but I'm trying to learn the 'right' way to make the grabber.  I saw that I could deduce the direct link but I was thinking that the right way to do it is to snoop out the link from their own pages.  That way if they make some small changes, a good grabber will try to get the video link dynamically.  But I guess for now that could work LOL.  However,  I am going to use your first suggestion since this is really about learning how to design/fix/patch a grabber.

 

As a crazy experiment, I am letting a few testers try the grabbers.  I designed a second upload page where my testers can experiment with the grabbers.  So far it has worked well, not sure if there is anything to worry about by allowing my users access to the grabbers.  It's also why I want to learn the design concepts.  If I'm going to make the grabbers user friendly and useful I have to know the design so I can keep them working.

 

 

Thanks,

Brian

 

PS is anyone still using avs forums? I get very few responses to my posts except, occasionally, from more experienced users like you @ theone and Dargre.  I thought people would like my tiny video webm movie preview mod but not a single reply LOL.  

 

Same with my user@mydomain.com craigslist style mod, not a single response LOL.  My users love the idea and I'm ready to go live with it, except for a small snag.  So either my ideas are not interesting or they are stupid or something, but I'd expect a few responses saying so, in a nice way, hopefully.

 

Also I get a big RED screen warning in chrome that this is a dangerous site, what is up with that?

 

Thanks again

Brian


Edited by brianatthebeach, 21 August 2017 - 03:09 AM.


#5 theone

theone

    Senior Member

  • Members
  • PipPipPipPip
  • 371 posts

Posted 21 August 2017 - 03:45 AM

The 'right way' is simply to pull the link(s) from the source and download them. With sites that use javascript to load content dynamically this can be more difficult.

 

Anytime a site makes changes you will likely need to adjust script manually anyway so no sense in wasting time trying to automate something like that. 

 

That's why the grabbers frequently break or require updates as it can never be predicted as to what changes will be made.