lsock/ntstat/NetworkStatistics

Questions and Answers about all things *OS (macOS, iOS, tvOS, watchOS)

lsock/ntstat/NetworkStatistics

Postby Domo » Tue Jan 12, 2016 11:02 pm

Hi Jonathan and everyone else!

I am digging deep into network statistics API on iOS 9. The low-level kernel socket way was my first approach, as it was used in the lsock utility, although I did have to add the "network.statistics" entitlement and update some structures. Then I turned to the "NetworkStatistics" private framework, because it seemed a much less hacky way of doing things (such as using different structs depending on kernel version). I got it to work without too much hassle, but then the real problems came.

First of all, I discovered that all this socket messaging seems unreliable: even using raw sockets didn't work every single time (tried on iOS 7 & 8). I used CFSocketXxx with callbacks, so I'm sure I didn't miss anything, but the kernel just didn't always reply with valid data from the first try. The only thing I could do is close and reconnect the goddamn socket, and everything went fine after that.

Now, using the NWStatisticsManager (from the NetworkStatistics framework) made things even worse. 80% time it works as expected and perfectly updates all data perpetually. 15% of launches it just goes blind: 0 sources (sockets) and nothing can recover this: reconnecting sockets, re-adding sources, destroying/recreating classes... 5% are strange too: it adds sources, but some (or all) of them have invalid data (null descriptions & counts). All this happens on iOS 9, and unfortunately I can't test it on iOS 7/8 (why did I update all my devices???) NWStatisticsManager uses OS_dispatch_queue-s internally, but I'm not sure how that's done. I only get resulting data in a tidy set of NSDictionaties.

I heard you will have a chapter dedicated to lsock in MOXii 2, maybe there's some secret stuff I don't know?

Thanks for a terrific book, and this whole site, and the articles, of course!
Domo
 
Posts: 9
Joined: Sun Aug 09, 2015 9:10 am

Re: lsock/ntstat/NetworkStatistics

Postby morpheus » Wed Jan 13, 2016 10:29 am

First, thanks for the kind words. It's nice people appreciate this. It's a lot of hard work.

Apple uses com.apple.network.statistics from what I see almost exclusively. To be honest, I haven't reversed the statistics frameworks, because as soon as I got what I needed off of ntstat it didn't make sense to invest time in those frameworks, especially with 400 or so other private frameworks which are FAR more important and even less documented.

Re: entitlements, yes. Those were added by Apple in OS X and iOS, and whereas in the latter I get by with faking them (benefit of bypassing code signing), the same cannot be done (easily) in 10.11. com.apple.network.statistics is actually now enforced in kernel mode (to my readers from 17.x.x.x, I say: WHY, People? WHY?! Focus on task ports first, will you?! Leave harmless statistics so my tools can work without hacks!). Changes are rather minimal, but that darn entitlement , yeah :x

MOXiI 2 will have loads more, which is why it's taking more (way longer) than I had expected. I'll also mention bandwidth accounting, and anything else I can reverse by then. Expect more updates soon.
morpheus
Site Admin
 
Posts: 531
Joined: Thu Apr 11, 2013 6:24 pm

Re: lsock/ntstat/NetworkStatistics

Postby Domo » Thu Jan 21, 2016 7:36 pm

Thanks for answering!

If I understood correctly, the ntstat entitlement is only enforced in the MAC kernel, and not on iOS?
I agree on Apple focusing on the wrong stuff, especially when it came to me that 17.x.x.x is the IP ;)

Thinking about the case of my problem, my first hypothesis was that the stats are already being collected by networkd, and it doesn't like servicing two or more clients. But, as the low-level ntstat.h api is working, i think it's just a problem with the framework. It's use is very limited in networkd, unlike my app.
Domo
 
Posts: 9
Joined: Sun Aug 09, 2015 9:10 am

Re: lsock/ntstat/NetworkStatistics

Postby morpheus » Fri Jan 29, 2016 3:04 am

SIP suddenly enforces a ton of entitlements which aren't so in iOS - making AMFI deny things which it normally turns a blind eye to (even in iOS 9.x). But don't worry, SIP is likely going to show up in iOS 10... :-(
morpheus
Site Admin
 
Posts: 531
Joined: Thu Apr 11, 2013 6:24 pm

Re: lsock/ntstat/NetworkStatistics

Postby sergio10 » Sun Mar 13, 2016 2:10 pm

Here is a version which works on El Capitan now: https://github.com/sotrosh/lsock/
sergio10
 
Posts: 5
Joined: Wed Dec 23, 2015 1:23 pm

Re: lsock/ntstat/NetworkStatistics

Postby backendbilly » Fri Mar 18, 2016 7:18 pm

@sergio10 thanks for making a working copy of lsock.

@J @sergio10 please try to make something clear to me

The assumption was (according to @J) that you need to have the proper entitlement "com.apple.network.statistics" plus having a valid certificate (not self-signed) used to sign lsock in order to have it working in OSX and iOS (not talking about disabling SIP here). With that limitation, I thought I could not use lsock anymore on iOS 9 without having a legit cert. With @sergio10 adding support for ElCapitan, it seems to work in both OSX and iOS 9 (thanks again @sergio10). Please explain to me why it now works without having the entitlement and cert in place.

Billy
backendbilly
Site Admin
 
Posts: 132
Joined: Fri May 29, 2015 5:58 pm

Re: lsock/ntstat/NetworkStatistics

Postby morpheus » Fri Mar 18, 2016 9:30 pm

I can explain here:

The entitlement is required for the network stats when SIP is enabled; The structures, too have changed. iOS doesnt enforce it, so it works. I'm in the process of updating my code (found a bug in it actually, too, sergio's derivative also has it and misses some connections). Expect something over the weekend.
morpheus
Site Admin
 
Posts: 531
Joined: Thu Apr 11, 2013 6:24 pm

Re: lsock/ntstat/NetworkStatistics

Postby backendbilly » Fri Mar 18, 2016 9:43 pm

I don't have any entitlements set on lsock in OSX, SIP is enabled, and it's working :?
backendbilly
Site Admin
 
Posts: 132
Joined: Fri May 29, 2015 5:58 pm

Re: lsock/ntstat/NetworkStatistics

Postby morpheus » Sat Mar 19, 2016 2:31 am

Yeah, I tried now on 10.11.2. Weird. The beta I had initially tried it on mandated the entitlement. Maybe AAPL got some common sense (or maybe some engineer lifted the entitlement? or they just don't enforce?)

Anyway, new and improved lsock coming soon.
morpheus
Site Admin
 
Posts: 531
Joined: Thu Apr 11, 2013 6:24 pm

Re: lsock/ntstat/NetworkStatistics

Postby abhirs » Fri Apr 07, 2017 2:00 am

morpheus wrote:Yeah, I tried now on 10.11.2. Weird. The beta I had initially tried it on mandated the entitlement. Maybe AAPL got some common sense (or maybe some engineer lifted the entitlement? or they just don't enforce?)

Anyway, new and improved lsock coming soon.

Hi Morpheus, Great and thanks a ton for the details. Please let me know is there any updated lsock available for 10.12? I am eagerly waiting to see my copy of "MacOS and iOS Internals, Volume III: Security & Insecurity"

Thanks.
abhirs
 
Posts: 10
Joined: Thu Apr 06, 2017 12:50 am

Next

Return to Questions and Answers

Who is online

Users browsing this forum: No registered users and 1 guest