In most cases, whether you have an obligation to withhold and pay payroll taxes will depend on whether your workers are properly classified as being employees or independent contractors.
Unfortunately, the standards used in distinguishing between employees, for whom you have payroll tax obligations, and independent contractors, for whom you do not, are highly subjective. So it wouldn't be at all unusual for a tax auditor to look at the same facts that you did and to reach an entirely different conclusion.
If you want to limit your risk of receiving a bill for unpaid payroll taxes because you misclassified a worker, we suggest that you always talk to your tax professional before you take the position that any of your workers are independent contractors. In close cases, you'd be well advised to take the added step of requesting an IRS determination as to a worker's proper classification. If the IRS is going to disagree with your classification, you can save a lot in tax penalties, interest, and legal costs if you find that out now rather than several years down the road.
How do you obtain an IRS determination? You request one by filing Form SS-8, Determination of Employee Work Status for Purposes of Federal Employment Taxes and Income Tax Withholding. You supplement the form with various information about your relationship with the worker (or class of workers) for whom you're requesting the ruling. For example, your request usually must include a copy of any agreement you may have with the worker, a detailed account of the services provided, and a description of how you supervise or direct the worker's services and pay for the services.