Advanced Site Analysis with Force-Directed Diagrams
•
20 recomendaciones•21,563 vistas
Analyze and diagnose SEO and other issues using Force Directed Diagrams. Yes, it's extra work. But it also helps make you understood.
Recomendados
Recomendados
Más contenido relacionado
Destacado
Destacado (19)
Más de Ian Lurie
Más de Ian Lurie (20)
Advanced Site Analysis with Force-Directed Diagrams
- 1. #SMX #25A @portentint Force-directed diagrams for teaching & analysis ADVANCED SEO VISUALIZATION
- 2. #SMX #25A @portentint Ian Lurie @portentint portent.com
- 3. #SMX #25A @portentint Slides at: http://portent.co/fd-slides
- 4. #SMX #25A @portentint I AM NOT A PROFESSIONAL
- 5. #SMX #25A @portentint A STORY AS OLD AS ME
- 6. #SMX #25A @portentint There is orphaned content. It’s important.
- 7. #SMX #25A @portentint how the @#!$(!! do I explain this?
- 9. #SMX #25A @portentint OUR JOB ISN’T JUST ABOUT BEING RIGHT
- 10. #SMX #25A @portentint OUR JOB IS ABOUT BEING UNDERSTOOD
- 11. #SMX #25A @portentint FORCE DIRECTED DIAGRAMS
- 12. #SMX #25A @portentint AS A TEACHING TOOL
- 14. #SMX #25A @portentint SHOW IT ORPHANED CONTENT
- 15. #SMX #25A @portentint “I GET IT”
- 16. #SMX #25A @portentint AS A DIAGNOSTIC TOOL
- 17. #SMX #25A @portentint I don’t wanna look through the log file!
- 19. #SMX #25A @portentint PAGINATION TUNNEL
- 21. #SMX #25A @portentint INFINITELY EXPANDING URLS hey handsome
- 22. #SMX #25A @portentint OUR JOB ISN’T JUST ABOUT BEING RIGHT
- 23. #SMX #25A @portentint OUR JOB IS ABOUT BEING UNDERSTOOD
- 25. #SMX #25A @portentint HOW-TO (IN 6 MINUTES)
- 26. #SMX #25A @portentint GRAVITY, REPULSION, EDGES
- 27. #SMX #25A @portentint NODE (URL) EDGE (LINK)
- 29. #SMX #25A @portentint REPULSIONget away!!!
- 31. #SMX #25A @portentint STEP 1: Create edges & nodes
- 32. #SMX #25A @portentint SCREAMING FROG
- 34. #SMX #25A @portentint SCREAMING FROG
- 35. #SMX #25A @portentint Source Destination CPU ABUSE
- 36. #SMX #25A @portentint DEDUPE EDGES
- 37. #SMX #25A @portentint GET NODE ‘WEIGHT’
- 38. #SMX #25A @portentint ADD IT TO THE EDGES SHEET
- 40. #SMX #25A @portentint THE SHEET http://portent.co/forcexls
- 41. #SMX #25A @portentint STEP 2: Generate your graph
- 53. #SMX #25A @portentint gaaah wtf
- 63. #SMX #25A @portentint THE GRAPH http://portent.co/gephi-pi
- 64. #SMX #25A @portentint THIS IS UGLY
- 65. #SMX #25A @portentint THIS IS NOT A SITE MAP
- 66. #SMX #25A @portentint CPU ABUSE
- 67. #SMX #25A @portentint < 4500 URLS
- 68. #SMX #25A @portentint 40000 URLS Ubuntu 6-core Xeon 128GB RAM 2x GPUs
- 69. #SMX #25A @portentint WHY DO IT?
- 70. #SMX #25A @portentint OUR JOB ISN’T JUST ABOUT BEING RIGHT
- 71. #SMX #25A @portentint OUR JOB IS ABOUT BEING UNDERSTOOD
- 72. #SMX #25A @portentint THANK YOU! SEE YOU @SMX WEST SAN JOSE, CA MARCH 1-3, 2016
- 73. #SMX #25A @portentint THANK YOU! SEE YOU @SMX WEST SAN JOSE, CA MARCH 1-3, 2016 Oh, also: Portent’s hiring SEOs, PPCs & all around smarty-pants
Notas del editor
- A few things: This presentation assumes you know how to use Excel and a basic web crawler. This is, after all, advanced. I’m not using particularly elegant tools. Fact is, this is ugly as hell. We’ve invested in building something ourselves because we got tired of melting computers.
- A few things: This presentation assumes you know how to use Excel and a basic web crawler. This is, after all, advanced. I’m not using particularly elegant tools. Fact is, this is ugly as hell. We’ve invested in building something ourselves because we got tired of melting computers.
- Why even use force-directed diagrams? They’re hard to generate. You could generate a more basic sitemap.
- Why even use force-directed diagrams? They’re hard to generate. You could generate a more basic sitemap.
- Somehow it feels… unnecessary, right? We’re just adding plumage when the information is already good.
- I’m just going to show you how to get data in and make it reasonably consumable
- Basically, you can configure how you display link counts, duplication, etc.
- Every force directed diagram needs nodes and edges. That’s each unique page, if you’re doing a website, and every link.
- Like this. More links = bigger circle, links increase ‘gravity.’ See how a page with one link is smaller, and the 1 link is further away? That makes this a valuable visualization tool above & beyond a standard sitemap.
- Like this. More links = bigger circle, links increase ‘gravity.’ See how a page with one link is smaller, and the 1 link is further away? That makes this a valuable visualization tool above & beyond a standard sitemap.
- So here’s how you get your serious geek on.
- First, we need a list of every URL on the site, and a list of all internal links to and from each URL. I recommend only doing all links and pages if you have a fast computer. Really, really fast. Really really really fast. More on that in a minute. Note: Filter the crap out of everything. Remove any page types you don’t want. In general, I find this whole process chokes if you have more than 5,000 links in your crawl. I use Screaming Frog to crawl the site. I filter out non-HTML stuff and non-200 responses. For my purposes here, that’s the right way to go.
- Then I export the crawl.
- Then I export a report: The ‘All Inlinks’ report shows me every link to and from every page. It’s a monster. It’s also what will give us the connections we need. It’ll become our edges and nodes. Delete all columns except Source and Destination. Then delete the https://www.portent.com or whatever. And now, one last pain in the ass. You only want unique links. So you have to filter in-place or use another tool. I use the advanced filter in Excel. Run it, then go for a walk or something. It takes a long time. Or, use a text editor or GREP or something to do it faster. (My secret – I actually use Sublime Text to do this – I just wanted to make fun of Excel) It is very important that the URLs in this sheet match the URLs in the other. Now, combine the data: Paste the contents of nodes.xls into a new tab in this sheet. Save it as ‘site-nodes-edges.xls’ or whatever.
- Don’t use tables. Your computer will melt.
- You’ll want to be able to pull the first link to any given page, ignoring the others. I just add a field called ‘first.’ Then add a field using the formula above. This is a convenience thing you’ll see in a minute.
- You’ll want to be able to pull the first link to any given page, ignoring the others. I just add a field called ‘first.’ Then add a field using the formula above. This is a convenience thing you’ll see in a minute.
- You’ll want to be able to pull the first link to any given page, ignoring the others. I just add a field called ‘first.’ Then add a field using the formula above. This is a convenience thing you’ll see in a minute.
- Export edges
- Export edges
- I use GEPHI. Note: There is a really annoying issue in Gephi relating to Java. Google ‘Gephi won’t start Java’ for solutions.
- Create a new project.
- Go to the Data Laboratory.
- Import spreadsheet
- Import edges
- Do the import. Set Connections to ‘Biginteger’
- Do the import. Check ‘create missing nodes.’ That makes it create the nodes for you, which is a lot easier.
- You should get something that looks like this
- Do the import. Check ‘create missing nodes.’ That makes it create the nodes for you, which is a lot easier.
- oookaaayyyyyy
- Now you can filter! Woot! Filter the edges table, removing ‘false.’
- That’ll get you just unique edges – remember?
- You’ll know it’s working because edges will be less than 100% visible
- I like to use Force Atlas layout. It’s up to you. You’ll want to play around to find the idea mix.
- Still not great, but improved.
- Prettier, using colors, messing w/ settings, etc. Something may not be right here.
- Clean it up to emphasize one or two things. Here, I’ve found a serious pagination tunnel. Reduce the number of edges and nodes for a little more clarity.
- Add some labels
- Yep
- Export edges
- This is a hideously ugly process. Learn it, then refine it and perfect it as much as you can.
- So here’s how you get your serious geek on.