Data-domain-cifs-and-nfs-troubleshooting-sg.pdf 601h4k

and (t port 53 or t port 88 or t port 135 or t port 137 or t port 138 or t port 139 or t port 389 or t port 445 or t port 464 or t port 543 or udp port 53 or udp port 88 or udp port 123 or udp port 137 or udp port 138 or udp port 139 or udp port 389) To use this filter with Wireshark, copy it from the student guide and paste it into the Wireshark capture filter field and provide the IP address for the Data Domain system.

Copyright 2015 EMC Corporation. All rights reserved.

Data Domain CIFS and NFS Troubleshooting

26

CIFS Capture Filters (Continued) The net tdump CLI command does not allow the use of capture filters, but the se tdump CLI command does. On the screen is an example of the filter being used in conjunction with the se tdump CLI command. Notice that the filter is enclosed by single quotes and is at the end of the command string. se tdump ‐i any ‐s 0 ‐w /ddvar/traces/cifs‐capture‐01.cap 'host 10.and (t port 53 or t port 88 or t port 135 or t port 137 or t port 138 or t port 139 or t port 389 or t port 445 or t port 464 or t port 543 or udp port 53 or udp port 88 or udp port 123 or udp port 137 or udp port 138 or udp port 139 or udp port 389)'



27

CIFS UDP and T Display Filters In Wireshark, the capture filters and display filters are defined using different (but similar) syntaxes. The display of a network trace may be limited to CIFS related traffic, by using the filter shown on the screen. Substitute the IP address of the target Data Domain system for the string, and cut and paste the filter directly into the Wireshark display filter field. ip.addr ==

&& (t.port == 53 || t.port == 88 || t.port == 135 || t.port == 137 || t.port == 138 || t.port == 139 || t.port == 389 || t.port == 445 || t.port == 464 || t.port == 543 || udp.port == 53 || udp.port == 88 || udp.port == 123 || udp.port == 137 || udp.port == 138 || udp.port == 139 || udp.port == 389)



28

LESSON SUMMARY - CIFS Troubleshooting Overview This lesson covered the following topics : •

Potential Problem Areas

•

High-level CIFS Troubleshooting Steps

•

Troubleshooting Network Connectivity

•

Troubleshooting CIFS Client

•

Troubleshooting the CIFS Server

•

CIFS Capture and Display Filters



29

Knowledge Check



30

LESSON - Troubleshooting Workgroup Authentication This lesson discusses the following topics: •

Reviewing CIFS Workgroup Authentication

•

Configuring CIFS Workgroup Authentication

•

CIFS Workgroup Connection Walkthrough



31

Reviewing CIFS Workgroup Authentication using CLI To review the CIFS Workgroup Authentication Configuration, use the cifs show config CLI command. This command displays the authentication mode - either Workgroup or Active Directory - as well as the parameters associated with the configured authentication method. The Workgroup name represents a logical grouping of devices that participate in a peer-to-peer Microsoft network. The Windows Internet Name Service (WINS) server is similar to the IP Domain Name Service (DNS). Whereas DNS provides the IP address based upon the IP name, WINS provides the IP address based upon the NetBIOS name. The NetBIOS (NB) host name allows the system to be addressed by NetBIOS clients. The IP host name will be used if an NB host name is not explicitly configured. The "net show hostname" CLI command provides you with the configured IP hostname. References •

About.com Workgroup http://compnetworking.about.com/cs/design/g/bldef_workgroup.htm

•

Workgroup (computer networking) http://en.wikipedia.org/wiki/Workgroup_(computer_networking)

•

NetBIOS Name Resolution http://technet.microsoft.com/en-us/library/cc958811.aspx

•

Windows Internet Name Service (WINS) http://en.wikipedia.org/wiki/WINS_Server



32

Reviewing CIFS Authentication - GUI Follow this process to review the CIFS Authentication configuration using the GUI: After selecting the target Data Domain system in the web-based GUI: 1. Select the "Data Management" menu item. 2. Select the "CIFS" sub menu item. 3. Select the "Configuration" tab. 4. View the configuration tab in the bottom portion of the screen.



33

Configuring CIFS Workgroup Authentication - CLI Use this process to configure the Data Domain system for CIFS workgroup authentication: 1. First, set the authentication mode to workgroup and provide the workgroup name using the "cifs set authentication workgroup" CLI command. The workgroup name may be up to 15 characters long. 2. Next, add s for the s that will access this device. These s can be used by the computers hosting backup applications. You will be prompted to configure a for the s. 3. The NB Hostname defaults to the first part of the IP host name ending at the first dot. If necessary, the NB Hostname may be changed from its default value by using the "cifs set nb-hostname" CLI command. Do not change the NB hostname from default values unless there is a specific need to do so. The NB hostname may be up to 15 characters long and can contain only alphanumeric , hyphen (), and underscore characters (_). Use the "net show hostname" CLI command to review the configured IP host name. 4. Finally, if needed, configure the WINS server using the "cifs set wins-server" CLI command. This command requires you to specify an IP address. Reference •


•


•


•




34

Configuring CIFS Workgroup Authentication using the web-based GUI After selecting the target Data Domain system from the left side of the web-based GUI, follow these steps to configure CIFS Workgroup Authentication: 1. First, select the "Data Management" menu item. 2. Next, select the "CIFS" sub menu item. 3. Now, select the "Configuration" tab. 4. Then, select the "Configure Authentication…" button. The "Configure Authentication" is displayed. 5. Select "Workgroup" from the mode pull down box. 6. Next, select the "General tab". 7. Now, select the "Use Default" checkbox if you wish to use the default workgroup name which is unspecified (blank). Unselect the "Use Default" checkbox if you wish to enter configure a Workgroup name. 8. Next, select the "Advanced Tab." 9. Now, select the "Use Default" server name if you wish to have the NB host name be derived from the IP host name. If you wish for the NB host name to be statically configured, you must unselect the checkbox and enter the name you prefer. 10. Finally, select "Ok" to accept the configuration changes and return to the CIFS configuration screen. If needed, the WINS server must be configured using the CLI. WINS server configuration is not available in the GUI.



35

Configuring CIFS Workgroup Authentication using the web-based GUI (Continued) After the CIFS workgroup authentication is configured, the CIFS has been added to the system. If the CIFS does not exist, add them now. Follow these steps to or configure the CIFS : 1. After selecting the target Data Domain system from the left side of the web-based GUI, select the "System Settings" menu item. 2. Select the "Access Management" sub menu item. 3. Select the "Local s" tab. 4. Examine the list of names and determine if the CIFS is already configured. 5. If the CIFS is not configured, select the create button. The "Create " configuration is displayed. 6. Select the "General" tab. 7. Enter the name, , and then the . 8. Select the role for the from the "Role" drop-down menu. The "data-access " role should be sufficient for the CIFS as they will not be required to ister the Data Domain system, only access the shared directories. 9. Select the "Advanced Tab." 10. If required, configure the " Aging Policy." 11. Next, enter the date for the to be disabled, if required. 12. Finally, select the "OK" button to accept the configuration changes and return to the configuration screen.



CIFS Workgroup Connection Walkthrough Workgroup initial authentication is a multi-step process 1. First, the client connects to CIFS server 2. Next the client negotiates the SMB dialect 3. Then, the client negotiates the authentication method 4. Now, the client connects to the named pipe 5. Finally, the client connects to shared resource The next few slides covers these steps in more detail.



37

Client Connects to Server 1. To initiate a workgroup authentication connection, the CIFS client verifies the target device is listening on T port 445 by sending a T SYN packet. If the target device is not listening on this port, the connection attempt times out or is refused. 2. If the server is listening on this port, it acknowledges the client by sending a SYN ACK packet from port 445. This is the standard response to a SYN request. 3. Finally, the CIFS client completes the 3-way T handshake by sending a T ACK to the target device on T port 445.



38

Client Negotiates SMB Dialect 1. The next thing the CIFS client must do is to negotiate the SMB dialect by sending an SMB Negotiate Protocol request to the server targeting T port 445. There are several versions of the SMB protocol and the client and server must agree on the version that will be used. The SMB Negotiate Protocol Request is used for this purpose. This packet provides the server with a list of the versions of the SMB protocol ed by the client. On screen is an example of the type of information provided to the server. 2. Next , the CIFS Server sends a negotiate protocol response. In the response is a version of SMB that the server selects from the list of SMB versions ed by the client.



39

Client Negotiates Authentication Method The next step in the process is to negotiate the authentication method that will be used. Through the years, the SMB protocol has made continuous improvements to its authentications schemes as market conditions change and technology improves. Because there are several authentication schemes, clients and servers need to agree which method to use. 1. First, the client sends a Session Setup Andx Request to tell the server the types of authentication it s. 2. The server responds by sending a challenge to the client. The challenge is a random number generated by the server. 3. Next, the client sends a Session Setup AndX Response to the server. To generate the response, the client uses the random number provided by the server against the provided by the operator and sends the result to the server along with its domain name, name, host name, and other information. 4. The server performs the same calculation against the in its database and then compares its result with the result provided by the client. The server then sends a Session Setup AndX Response packet to the client to let it know if the client has successfully authenticated. The Data Domain system does not distinguish between an incorrect name or an incorrect . A trace of a transaction with a Data Domain system shows that the system responds with the SMB error STATUS_LOGON_FAILURE. All STATUS_LOGON _FAILURES are interpreted by the client as bad , even if the does not exist on the server.



40

Client Connects to Named Pipe The named pipe service (IPC$) enables the client to issue Remote Procedure Calls (RPCs) to the server. Among other things, the CIFS client uses the named pipe mechanism to: •

List all shares

•

List all s

•

List files within a share

•

Stop/Start services

1. First , the client attempts to connect to the named pipe by using the Tree Connect AndX Request packet along with the path: \\ \IPC$ 2. The server send a success or failure message to the client using the Tree Connect Andx Response packet.



41

Client Connects to Share The client has now successfully gained access to the server. The next thing to do is to actually gain access to the shared resource. Access to resources may be restricted based upon the or the name or IP address of the client workstation. The client follows this process to determine if the may access the share: 1. The client requests access to resource by using a Tree Connect Andx Request packet along with the target path (\\ \<share-name>). 2. Server verifies access based upon the rights configured with the sharename. The server sends the client its response using a Tree Connect Andx Response packet.



42

LESSON SUMMARY - Troubleshooting Workgroup Authentication This lesson covered the following topics: •

Reviewing CIFS Workgroup Authentication

•

Configuring CIFS Workgroup Authentication

•

CIFS Workgroup Connection Walkthrough



43

Knowledge Check



44

LESSON - Troubleshooting Active Directory Authentication This lesson covers the following topics: •

Preparing for CIFS Active Directory Authentication

•

Reviewing CIFS Active Directory Authentication Configuration

•

Configuring Kerberos / AD Authentication



45

Preparing for CIFS Active Directory Authentication There is some information you need to gather as you prepare to configure Active Directory authentication on the Data Domain system. This information includes: •

The domain name to which the Data Domain system will be a member. The domain name is also referred to as the realm name.

•

If you going to use manual entries, you need the DNS name or IP address of the closest domain controller. You don't need this information if you are going to configure the Data Domain system to automatically locate a domain controller. When the Data Domain system is configured to automatically locate the Domain Controller (DC), it uses the first DC to respond to its query.

•

You'll need the name for the that the Data Domain system will use to interact with the Domain Controller.

•

This should be configured to allow the Data Domain system to read and write to the Active Directory to provide updated information on shared resources or changes in permissions and s on the Data Domain system.

You will also need the for the Data Domain system's . If the Active Directory and has been configured correctly, this information should enable the Data Domain system to "" the domain. References •

181329 : Using CIFS "Set Authentication Active-Directory" Command https://.emc.com/kb/181329

•

181313 : ing a Data Domain System to a Windows Domain https://.emc.com/kb/181313

•

180810 : CIFS Content Browsing Page https://.emc.com/kb/180810



46

Reviewing CIFS Active Directory Authentication using CLI To review the CIFS Authentication Configuration, use the cifs show config CLI command. This command displays the authentication mode - either Workgroup or Active Directory - as well as the parameters associated with the configured authentication method. The Mode field should be set to Active-Directory, not Workgroup. The Realm field is the same as the Domain Name.

that it is configured as expected.

The Domain Controllers field shows the DCs currently being used by the Data Domain system. the Windows Internet Name Service (WINS) server field. WINS is similar to the IP Domain Name Service (DNS). DNS provides the IP address based upon the IP name and WINS provides the IP address based upon the NetBIOS name. the NetBIOS (NB) Hostname field is configured correctly. The NB hostname allows the system to be addressed by NetBIOS clients. The default NB name is the first 15 characters of the IP host name. Use the net show hostname CLI command to view the configured IP hostname. References •


•


•


•




47

Reviewing CIFS Authentication - GUI Follow this process to review the CIFS Authentication configuration using the GUI: After selecting the target Data Domain system in the web-based GUI: 1. Select the "Data Management" menu item. 2. Select the "CIFS" sub menu item. 3. Select the "Configuration" tab. 4. View the configuration tab in the bottom portion of the screen.



48

Configuring CIFS Active Directory Authentication - CLI After gathering the information needed to configure Active Directory authentication, follow these steps: 1. First, set the authentication options. A. If you are using the Window DC to provide Kerberos authentication for your NFS clients, use the authentication kerberos CLI command.

B. If NFS Clients do not use the Windows DC for Kerberos Authentication, use the "cifs set

authentication active-directory" CLI command. This command requires the realm (domain name) and IP addresses of the domain controllers as arguments. If you wish for the Data Domain system to discover the IP addresses of the domain controllers, substitute an asterisks (*) for the domain controller's IP address.

2. After invoking the command to set the authentication options, the Data Domain system disables CIFS, prompts you for the name and , and attempts to "" the domain by locating and authenticating with the Active Directory on the Domain Controller. The Data Domain system will let you know if this step is successful or not. 3. Next, If required, you should set the NetBIOS host name using the "cifs set nb-hostname" CLI command. Usually this step is not needed. 4. Also, if there are devices in your network that require the services of a WINS server to interact with the Data Domain system, you should configure the WINS server IP address using the "cifs set winsserver" CLI command. In most environments, this step is unnecessary.



49

Configuring CIFS IPv6 AD Authentication - CLI Starting with DD OS 5.5.1, IPv6 addresses are allowed with the cifs set authentication CLI command. There is an example on the screen.



50

Configuring CIFS AD Authentication With NFS - GUI After selecting the target Data Domain system from the left side of the web-based GUI, follow these steps to configure Kerberos Authentication for CIFS and NFS clients using the web-based GUI: 1. First, select the "System Settings" menu item. 2. Next, select the "Access Management" sub menu item. 3. Now, select the "Authentication" tab. 4. Review the "Active Directory" settings. 5. Select "Configure" to change the configuration. The Active Directory / Kerberos settings is displayed. 6. Next, select the "Windows / Active Directory" option from the screen. 7. Now, select "Next". 8. Enter the Realm or Domain name and the credentials required for the Data Domain system to the domain. 9. Select "Next." 10. Next, configure the "CIFS server name." 11. Now, select to automatically assign or manually configure the IP addresses for the Domain Controllers. If you choose to manually configure the DC's IP addresses, enter them at this time. 12. Next, choose whether to use the default Organizational Unit "Computers" or to specify another. 13. Select "Next". 14. Review the summary configuration information. 15. Select "Finish" accept the configuration. References •

Wikipedia - Organizational Unit http://en.wikipedia.org/wiki/Organizational_Unit



51

Configuring CIFS AD Authentication Without NFS - GUI After selecting the target Data Domain system from the left side of the web-based GUI, follow these steps to configure CIFS Active Directory Authentication using the web-based GUI: 1. First, select the "Data Management" menu item. 2. Next, select the "CIFS" sub menu item. 3. Now, select the "Configuration" tab. 4. Select the "Configure Authentication…" button. The "Configure Authentication" is displayed. 5. Next, select "Active Directory" from the mode pull down box. 6. Now, select the "General tab". 7. Enter the Realm or Domain name in the input box. 8. Enter the credentials required for the Data Domain system to the domain. 9. Select the "Advanced Tab." 10. Next, select the "Use Default" server name if you wish to have the NB host name be derived from the IP host name. If you wish for the NB host name to be manually configured, you must unselect the checkbox and enter a name. 11. Now, select to automatically or manually configure the IP addresses for the Domain Controllers. If you choose to manually configure the DC's IP addresses, enter them at this time. 12. Next, choose whether to use the default Organizational Unit "Computers" or to specify another by deselecting the "Use Default Computers" checkbox. Enter the name of the "Organizational Unit" to which the Data Domain system will belong. 13. Finally, select "Ok" to accept the configuration changes and return to the CIFS configuration screen. If needed, the WINS server must be configured using the CLI. WINS server configuration is not available in the GUI. References •

Wikipedia - Organizational Unit http://en.wikipedia.org/wiki/Organizational_Unit



52

LESSON SUMMARY - Troubleshooting Active Directory Authentication This lesson covered the following topics: •

Preparing for CIFS Active Directory Authentication

•

Reviewing CIFS Active Directory Authentication Configuration

•

Configuring Kerberos / AD Authentication



53

Knowledge Check



54

LESSON - Troubleshooting Domain Issues This lesson discusses the following topics: •

Describing the term " Domain Issue"

•

Describing the Domain Transaction Flow

•

Listing the reasons for using the command line to configure Active Directory Authentication

•

Addressing Domain Controller Not Found issues

•

Addressing other common problems such as: Time skew, Invalid , invalid , and AD configuration



55

Describing Domain Issues The Data Domain system must the domain (realm) in to interact with the Active Directory. For this to happen, the Data Domain system must be able to: •

Locate the Domain Controller (DC). This requires that the Data Domain system be able to resolve the Domain name for the controller and to be able to transmit data using the required UDP and T ports.

•

to the DC. This means the Data Domain system must have an on the DC.

•

Read and Write to the Organizational Unit (OU) on the Active Directory (AD). This means that the OU on the AD needs to allow the Data Domain system to manipulate the contents.

•

Authenticate using Kerberos

•

Keep time in sync with DC.



56

Domain - Transaction Flow The follow high-level steps outlines the individual transactions executed when a Data Domain system s a domain. 1. First, the Data Domain system looks up the DNS name for the DC. 2. If the last step was successful, the Data Domain system sends a Connectionless Lightweight Directory Access Protocol (CLDAP) query to the DC to determine if the DC is running LDAP. CLDAP uses UDP instead of T, so it takes less time to setup and execute the query. 3. Once the Data Domain system determines the DC is running LDAP, the Data Domain system initiates a Kerberos (KRB) ticket exchange. 4. Next, the Data Domain system sends an LDAP query to the DC to determine if the OU exists and if the Data Domain system object already exist in the OU. 5. If the Data Domain system object does not exist in the OU, the Data Domain system will create the object, provided it has sufficient permissions to do so. Reference •

Connection-less Lightweight X.500 Directory Access Protocol https://tools.ietf.org/html/rfc1798

•

Kerberos: The Network Authentication Protocol http://web.mit.edu/kerberos/



57

Domain - Packet Trace On the screen is a walkthrough of a capture of a packet trace for a Data Domain system ing a domain. This trace should help you in understanding how the Data Domain system s the domain, and to understand what to look for when assisting customers. Capture the Trace Before you can examine a trace you must first capture it. There are a number of ways to accomplish this, but the easiest is to use the net tdump capture CLI command. The net tdump capture CLI command places the capture in the /ddvar/traces directory on the Data Domain system. You may also capture a trace using Wireshark on the Domain Controller. The trace can be reviewed using tshark (from Wireshark) or ethereal. DNS Lookup The first thing that happens is the Data Domain system requests the IP address of the DC from the DNS. You can mimic this transaction with the net lookup command. CLDAP Authentication The next transaction is the CLDAP authentication. These packets are used to determine if DC is running LDAP. Kerberos Authentication The next transaction is the Kerberos authentication request and response. Reference •

TDUMP Man Page http://www.tdump.org/manpages/tdump.1.html

•

Tethereal Man Page http://www.linuxcommand.org/man_pages/tethereal1.html



58

Domain - Packet Trace (SE Mode Tools) •

The tdump utility is also available in SE mode and in Bash mode. In these modes, it offers more options if needed.

•

Saving the tdump to the CIFS directory ensures the capture will be included in any bundles generated afterwards.

•

You may review the trace on the Data Domain system using the tethereal utility in SE-mode or bashmode.

• •

The dash lowercase r option (-r) identifies the capture file to read. The dash uppercase R option (-R) identifies the types of packets to display. As you can see, SMB, LDAP, DNS, CLDAP, and Kerberos packets are to be displayed.



59

Domain - Packet Trace (Continued) •

Now you'll see the SMB negotiation.

•

The next step is to setup an SMB session with the DC.

•

Following that, the Data Domain system connects to the IPC share on the DC.

•

Now, connect to the lsass service which handles the LDAP protocol.

References •

Local Security Authority Subsystem Service (LSASS) http://msdn.microsoft.com/en-us/library/aa939478(v=winembedded.5).aspx

•

Security Subsystem Architecture http://technet.microsoft.com/en-us/library/cc961760.aspx

•

Understanding LDAP Security Processing http://blogs.technet.com/b/askds/archive/2009/09/21/understanding-ldap-security-processing.aspx

•

About IPC$ http://.microsoft.com/kb/314984 http://msdn.microsoft.com/en-us/library/windows/desktop/aa365574(v=vs.85).aspx



60

Domain - Packet Trace (Continued) The next section shows the Data Domain system sending a Query to the DC using LDAP to obtain the base. Finally, the Data Domain system attempts to add itself to the directory on the DC. If an entry exists, the DC sends an entryAlreadyExists message.



61

Domain - Packet Trace (Continued) When working with CIFS-related -domain network traces, the packets follow the steps for ing a Windows domain. If certain steps keep repeating, they need to be investigated.



62

Domain- Use the Command Line When configuring Active Directory authentication, you should use the CLI because of the it provides, the ready access to log files, and the troubleshooting tools available to help you diagnose issues. domain issues quite often manifest themselves when the system first configures Active Directory authentication. When using the "cifs set authentication active-directory " command, the Data Domain system often provides to indicate the possible cause of the problem as well as some steps that can be taken to fix the issue. On the screen is the error message that is displayed when the system cannot find the domain controller. The message reminds the system to the DC can be found through the DNS and that port 389 is not blocked by the firewall. Not only can domain issues occur during the initial configuration period, but also when the Data Domain system attempts to periodically resync with other network components. Notes Error from /ddr/var/log/debug/messages. engineering sms: NOTICE: cifs_: result_status [3] result_string [Failed to lookup the domain controller for given domain. Check that the domain name is correctly entered. Check that your DNS server is reachable, and that your system is configured to use DNS in nsswitch. Check that port 389 UDP is not blocked by your firewall.] Error from /ddr/var/log/debug/cifs/_domain.log 20140507130333:ERROR:Lsass Error [CENTERROR_DOMAIN_UNRESOLVED_DOMAIN_NAME]



63

Domain - DC Not Found If the Data Domain system cannot find the DC, a message will be entered in the "messages.engineering " log file with the text "Failed to lookup the domain controller for given domain." Windows Client Environment Verification You can find out more about the domain by querying a client that is supposed to be in the same domain and has successfully logged into the network. Using the set command in a command window provides you with a list of environment variables including the computer name, the server (which is the same as the domain controller), the DNS domain name (which is the domain name in FQDN format), the domain name, and the name. Compare this information to what you are using to configure Active Directory on the Data Domain system and ensure any differences are understood.



64

Domain - DC Not Found (Continued) Windows Client DNS Verification Next check the operation of the DNS from the windows client. First, get a list of domain controller IP addresses for the realm by using the nslookup " " command. Specify the realm in fully qualified domain name format. If you've been given an IP address instead of a DNS name for the DC, make sure the IP address is listed in the output. If you've been given the DNS name of DC, lookup the IP address by using the "nslookup " command on the windows workstation. When the IP address returns, check to see if it is on the list of DCs for that domain. You can retrieve the name DC by using the "nslookup " command on the windows client. Finally, you may wish to confirm the DDR's DNS entries by using the "nslookup" command with the Data Domain system host name and IP address as arguments. Data Domain system DNS Verification You can the DDR's access to the realm's DNS entry by using the "net lookup" CLI command. the Data Domain system can obtain the realm information from the DNS by using the "net lookup" command. Specify the realm name in fully qualified DNS name (FQDN) format. Obtain the IP address of the DC by invoking the "net lookup" CLI command with the DC's fully qualified DNS name as the target. Obtain the DC's fully qualified DNS name by invoking the "net lookup" CLI command with the DC's IP address as the target.



65

Domain - DC Not Found (Continued) Connectivity Verification and Latency Assessment Use the ping command to connectivity and latency. From the windows client, ping the Data Domain system and the DC. From the Data Domain system, ping the DC. Note, the ping command may fail if ICMP ECHO packets are filtered by firewalls in the network or if the target device is configured to ignore these packets. Do not assume a ping failure indicates that the device is down or there is no connectivity. A failure with PING only means that PING was not able to connectivity and more investigation is required. References •

Wikipedia Environment Variables http://en.wikipedia.org/wiki/Environment_variables

•

181313 : ing a Data Domain System to a Windows Domain https://.emc.com/kb/181313

•

Microsoft - using nslookup http://.microsoft.com/kb/200525



66

Domain - DC Not Found (Continued) If a CIFS client cannot map to a Data Domain System in an environment with multiple domain controllers, the ing of the Data Domain System to an active directory domain may have failed. Out of sync domain controllers can cause the problem. Look in the Data Domain System log files for messages similar to "Preauthentication" failed or "Client not found in Kerberos database." The workaround is to re- the Data Domain System to a domain using a single IP address in the CIFS command on the Data Domain System.



67

Domain - DC Not Found (deep dive) You can get the name of a domain controller (and a lot of other useful information) by entering bash mode and executing the "lw-get-dc-name" command. An example of this output is available in the student guide. This command is part of the likewise suite and is found in the /opt/likewise/bin directory on the Data Domain system. If this command is successful, it returns the name of the domain controller. Ping the controller to accessibility. If the "lw-get-dc-name" command is unable to obtain information about the realm , it returns an error. You can workaround this issue by using the IP address of the DC instead of the DC's FQDN. The fields returned are named using Hungarian notation. The names listed are prepended with dw, w, psz, and puc to indicate the type of variable in source code. They are (d)ouble (w)ord, (w)ord, (p)ointer to (s)tring ending in (z)ero, and (p)ointer to (u)nsigned (c)har. I'm telling you this so you don't waste time looking up the meaning yourself. References • Likewise Enterprise Installation and istration Guide (pdf) http://one.emc.com/clearspace/docs/DOC-97020 • Likewise Open Installation and istration Guide http://one.emc.com/clearspace/docs/DOC-97021 • Hungarian notation. http://www.cse.iitk.ac.in/s/dsrkg/cs245/html/Guide.htm Example Output from Successful Completion of the "lw-get-dc-name" Command lw‐get‐dc‐name corp.emc.com Printing LWNET_DC_INFO fields: =============================== dwDomainControllerAddressType = 23 dwFlags = 12796 dwVersion = 5 wLMToken = 65535 wNTToken = 65535 pszDomainControllerName = corpcascv1.corp.emc.com pszDomainControllerAddress = 137.69.224.15 pucDomainGUID(hex) = E9 07 84 1F E1 F6 68 49 8D C6 8E 3B AE 94 8F 1D pszNetBIOSDomainName = CORP pszFullyQualifiedDomainName = corp.emc.com pszDnsForestName = emcroot.emc.com pszDCSiteName = CorpUSCASantaClara1 pszClientSiteName = CorpUSCASantaClara1 pszNetBIOSHostName = CORPCASCV1 pszName = <EMPTY>



68

Domain - Time Difference There are some time sync requirements in order for active directory authentication to work. These requirements are that the time on the Data Domain system and DC be within 5 minutes and that the Data Domain system and DC be in the same time zone. You can the Data Domain system's time and timezone configuration by using the system show date and config show timezone CLI commands. If needed, you can set the date and time zone with the system set date and config set timezone CLI commands. If the you are using a Network Time Protocol (NTP) server, use the ntp status and ntp show config CLI commands to the NTP settings on the Data Domain system. that NTP uses port 123. Communication between the Domain Controllers, Data Domain systems, and NTP servers should not be blocked by firewalls.



69

Domain - DC Time in Bash You can the time on the Domain Controller from the Data Domain system by using the lw-get-dctime BASH command. This command is located in the /opt/likewise/bin/directory.



70

Domain - Invalid If one of the following error messages appears in the messages.engineering or _domain.log file after executing the "cifs set authentication active-directory" CLI command, the name may not be configured correctly on the DC: •

Failed to the domain with the following error message: *** Permission denied

•

kinit(v5): Preauthentication failed while getting initial credentials

•

The is invalid or the is incorrect for the given name

•

The does not exist or syntax (domain\) is incorrect.

the name exists in target domain by checking the DC. with the Down Level Logon Name (domain\name) format using the domain prefix if the is a regular domain . Employ the Principal Name (UPN) style if the is a trusted domain . , names are not case sensitive while s are. References •

Name Formats http://msdn.microsoft.com/en-us/library/windows/desktop/aa380525(v=vs.85).aspx



71

Domain - Incorrect Error messages in the messages.engineering or _domain.log can identify that the problem lies with the employed by the .



72

Domain - Unauthorized A "permission denied" error message can be displayed after executing the cifs set authentication active-directory CLI command or written to the messages.engineering or _domain.log. This may indicate that the Data Domain system name does have permission to write to the Organizational Unit (OU) configured on the Data Domain system. There are a few things you an try to address this issue: •

First, the Data Domain system has permission to write to the OU by looking at the Domain Controller.

•

Next, you can try logging into the DC from a Windows client and the Data Domain is able to write a new object to the OU. If a new section is not available, it indicates the Data Domain system does not have permissions.

•

You can also configure the Data Domain system to use the credentials of a whose access is confirmed to work.



73

Domain - Authentication Issues •

authentication issues will manifest themselves by presenting an error when the "cifs config authentication active-directory" CLI command is executed or by writing the error to the clients.log or the cifs.log files.

•

If you suspect an authentication error, the cifs feature is enabled and running by using the cifs status CLI command.

•

Next, you can check the configuration of the "idmap-type" and "ntfs-acls" options. The "idmap-type" option should be set to "rid, "none", or not set. The "ntfs-acls" option should be "enabled" or not set.

•

Now, the NB-hostname is set and that the name has a maximum of 15 characters.

•

Finally, you should:

• the authentication type with the "cifs show config" CLI command. • View the s types with the " show list" CLI command. • the Data Domain system configured on the system with the "cifs troubleshooting -list" CLI command.

• View the details of the Data Domain system cifs with the "cifs troubleshooting " CLI command.



74

Domain - Authentication Issues (Continued) •

You can also go into SE-mode to the status of the processes used for authentication.

•

Execute the se ps -A SE-mode command and look for lsassd, lwiod, and netlogond processes.



75

Diagnosing AD Config When diagnosing the AD, the NetBIOS name matches the first part of the DNS name. Also, if strong security policies are set on the AD, make sure that anonymous access to the IPC$ share is allowed and that the NETLOGON pipe is enabled.



76

LESSON SUMMARY- Troubleshooting Domain Issues This lesson covered the following topics: • Describing the term " Domain Issue" • Describing the Domain Transaction Flow • Listing the reasons for using the command line to configure Active Directory Authentication • Address Domain Controller Not Found issues • Addressing other common problems such as: Time skew, Invalid , invalid , and AD configuration



77

Knowledge Check



78

LESSON - CIFS Authentication Troubleshooting This lesson discusses the following topics: • • • • •

Using trusted domain s The loss of the Data Domain system machine Kerberos ticket decryption error IDMAP issues SSH to Data Domain system not working



79

Authentication - Trusted Domain s When troubleshooting the authentication of trusted domain s, the authentication mode for the Data Domain system using the "cifs show config" CLI command. You should also determine which authentication method is in use as well as the 's location. Determine if the has an that is configured on the local Data Domain system, or is the 's is part of the same domain as the Data Domain system, or is the 's is part of a trusted domain.



80

Authentication - Trusted Domain s (Continued) If necessary, use the se ps -A SE-mode command to that the winbindd or lsassd processes are running. The winbindd process is used by Samba and the lsassd process is used by Likewise.



81

Authentication - Trusted Domain s (Continued) If the is from a trusted domain, use the "cifs option show" CLI command to the CIFS allowtrusteddomains option is enabled on the Data Domain system. If it is not enabled, you can use the "cifs option set allowtrusteddomains enabled" CLI to enable the option. Review the client log file on Data Domain system. The client log file is named using the IP address of the client with the .log extension (192.168.50.99.log). This file may indicate that there is a memory allocation error or the client connection failed because of too many sessions. The client log file is located at /ddvar/log/debug/cifs. # cifs option show Currently Set Options: "idmap‐type" is set to "rid" "ntfs‐acls" is set to "enabled" Known Useful Options: allowtrusteddomains enabled | disabled (default: enabled) loglevel [0‐10] (default: 1) maxxmit                    [16384‐65536] (default: 65536) restrict‐anonymous         enabled | disabled (default: disabled) smbd‐mem‐limit             [52428800‐1073741824] (default: 209715200) ntfs‐acls enabled | disabled (default: enabled) idmap‐type                 none | rid (default: rid) organizational‐unit        e.g. "Computers/Servers/ddr units" (default: Computers) dd group1            e.g. "domain\group2" (default: "Domain s") dd group2            e.g. "domain\group1" (default: None) dd group1             e.g. "domain\group1" (default: None) dd group2             e.g. "domain\group2" (default: None) dd backup‐operator group1  e.g. "domain\group1" (default: None) dd backup‐operator group2  e.g. "domain\group2" (default: None)



82

Authentication - Loss of Data Domain system Machine If the Data Domain system has lost its machine on the Active Directory, check for errors in the "winbindd.log". Use the "wbinfo" client command to confirm that the Data Domain system is disconnected from the domain. Re the Data Domain system to the domain.



83

Authentication - Kerberos Ticket Decryption Error This error is caused when the Windows client attempts to use a stale Kerberos ticket. This can happen when the Data Domain System is rebooted, or re-s a domain, and obtains a new Kerberos ticket. The existing ticket on the Windows client will then be stale and not work. This can also happen if the Data Domain system changes its on the domain controller. This change in s can render the ticket on the client as invalid. This error can also be caused when there is a discrepancy of greater than 5 minutes between system times on the client, Data Domain system, or domain controller. You can work around this issue in one of the following ways: • Logoff and log back on to the client. This causes the client to obtain a new ticket from the Kerberos server. • Use the klist utility to list and purge stale Kerberos tickets from the Windows client. • The klist utility was part of the Windows Resource Kit Tools for Windows 2003, and is available for other Windows platforms. • Type klist at the command prompt to see if it is installed on your windows client. • Reboot the windows client if the problem persists. Example C:>klist Current LogonId is 0:0xFFFFF Cached Tickets: (1) #0>     Client:  @ CORP.EXAMPLE.COM Server: krb5/CORP.EXAMPLE.COM @ CORP.EXAMPLE.COM KerbTicket Encryption Type: AES‐256‐CTS‐HMAC‐SHA1‐96 Ticket Flags 0x40e00000 ‐> forwardable renewable initial pre_authent Start Time: 9/17/2015 10:42:41 (local) End Time:   9/17/2015 20:42:41 (local) Renew Time: 9/24/2015 10:42:41 (local) Session Key Type: AES‐256‐CTS‐HMAC‐SHA1‐96



84

Domain - Kerberos Ticket Decryption Error (Continued) If needed, you can empty the Kerberos cache on the Data Domain system by using the "lw-ad-cache -delete-all" bash command. This command is located in the /opt/likewise/bin directory.



85

Authentication - IDMAP Issue The IDMAP facility is the part of the CIFs process that maps UNIX Identifiers (UIDs) and Group Identifiers (GIDs) to Windows security identifiers (SIDs). In a multi-protocol environment, there is a potential for IDMAP issues where the system is not able to associate the UIDs, GIDs, and SIDs with one another. In earlier versions of DD OS, this type of error created a Relative Identifier (RID) error message in the log.winbindd-idmap or log.wb* files. These files may be found in the /ddvar/log/debug/cifs directory on the Data Domain system. Use the log list debug CLI command to the existence of these log files. Use the log view CLI command to display the log files. Reference • Chapter 14. Identity Mapping (IDMAP) https://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/idmapper.html • ID Man Page http://ss64.com/bash/id.html



86

Authentication - IDMAP Issue If you suspect the IDMAP facility is having issues, examine the various IDs associated use by executing the cifs troubleshooting CLI command on the Data Domain system. The cifs troubleshooting CLI command provides the UID, GID, SID, name, and group names associated with the . An example of the output from this command is shown on the screen. # cifs troubleshooting  <name>        ddsys\sys ID    100 SID        5‐1‐5‐21‐3273096021‐3160076067‐2061037076‐1200 Group      ddsys\DD s Group ID   60001800 id         uid=100(ddsys\sys) group=60001800(ddsys\DD s) groups=60001800(ddsys\DD s)



87

Authentication - IDMAP Issue (Continued) If the cifs troubleshooting CLI command does not provide adequate information, or is not available on the Data Domain system, the id <name> bash command can be used. This command allows you to directly interrogate the IDMAP facility on the Data Domain system. The output of this command should provide the various UNIX IDs associated with the . On screen is an example of the output of this command. !# id sys id=100(sys) gid=50() groups=50(),60001800(ddsys\DD s)



88

Authentication - IDMAP Issues (Continued) If the issue involves a connected Windows client, the 's Windows 's SID by executing the whoami program on the Windows client. An example of the output from this command is shown on the screen.

C:> whoami / INFORMATION ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ Name    SID ============ =============================================== corp\bkup01  S‐1‐5‐21‐854245398‐1972579041‐362288127‐1234567 Another program, the Windows Management Instrumentation Command-line (WMIC), also provides the SID. Unfortunately, this Windows utility takes longer to execute than the whoami program. To display the 's SID using the wmic, use the syntax shown on the screen:

C:> wmic where name='%name%' get sid SID S‐1‐5‐21‐854245398‐1972579041‐362288127‐1234567 Reference • Get SID of http://www.windows-commandline.com/get-sid-of-/ • WMIC - Take Command-line Control over WMI http://technet.microsoft.com/en-us/library/bb742610.aspx http://msdn.microsoft.com/en-us/library/aa394531(v=vs.85).aspx



89

Authentication - SSH to Data Domain system not Working If the customer would like to SSH into DD system using windows , check CIFS authentication is enabled for access. Then review that IDMAP is working for the . If the customer is unable to create an SSH connection to the Data Domain system from a Windows client, that CIFS authentication is enabled for access by executing the "access authentication show" CLI command. # access authentication show CIFS authentication: disabled If CIFS authentication is disabled, enable it with the "access authentication add cifs" CLI command. # access authentication add cifs CIFS authentication:    enabled Review that the UIDs is working for the : # cifs troubleshooting  sys         bluewhale\sys ID     100 SID         5‐1‐5‐21‐3273096021‐3160076067‐2061037076‐1200 Group       bluewhale\DD s Group ID    60001800 id          uid=100(bluewhale\sys) group=60001800(bluewhale\DD s) groups=60001800(bluewhale\DD s)



90

LESSON SUMMARY - CIFS Authentication Troubleshooting During this lesson the following topics were covered: • Using trusted domain s • The loss of the Data Domain system machine • Kerberos ticket decryption error • IDMAP issues • SSH to Data Domain system not working



91

Knowledge Check



92

LESSON - Share and File Access Control Troubleshooting This lesson discusses the following topics: • Troubleshooting Share-Level Access Control Issues • Describing and Troubleshooting SMB g • Troubleshooting File-level Access Control Issues



93

Share Authorization The following is a high-level overview of how CIFS authentication and authorization works: 1. First, the client establishes a connection to the file server - or Data Domain system in this case- and the file server es information about itself back to the client. 2. Now, the server determines if the exists by looking in its local database, or by sending a query to the Active Directory. If the exists, the Data Domain system requests the Windows SID for every group associated with the . 3. The next step is to authenticate the by determining if the provided the correct or other credentials. This can be done by interrogating the Active Directory or by looking at the local s on the Data Domain system. There will be a message in the clients.log file to indicate success or failure. 4. Now, the Windows client connects to the IPC share on the Data Domain system. If the Windows client fails to connect to the IPC share, it assumes that the Data Domain system is not configured as a CIFS server. 5. If the connection to the IPC share is successful, the Windows client verifies the share exists by attempting to mount the share with an SMB Tree Connect to the Data Domain system. 6. Next, the Data Domain system verifies that the is allowed to access the share. 7. Finally, the Data Domain system verifies that the machine (also known as the client) is allowed access to share.



94

Share Access - istrative Restrictions Sometimes the cannot access the Windows shared folder because of intentional or unintentional istrative restrictions. The restrictions can be: Client access restrictions which identify the hosts from which s are allowed to access the shared folder. Group access restrictions which identify the groups whose are allowed access to the shared folder. access restrictions which identify the s who are allowed access to the shared folder. Host Access Restrictions Use the "cifs share show <sharename>" CLI command to see if the shared folder has client restrictions applied. When creating client restrictions, be aware that all clients are restricted by default when the share is created. You can specify which clients will be allowed to access the shared folder when you create the folder, or you can modify the client list as a subsequent step. If you choose to restrict access to particular clients, you must provide the IP address or the DNS name of individual stations. This means that you cannot restrict access to a group of clients with a single entry. If you use a DNS host name in the client list, the Data Domain system must be able to resolve that host name when that client attempts to access the shared folder. / Group Restricted Access Access may be restricted to certain s or groups. As long as the access list is empty, all s are allowed access to the shared folder. If there is any entry in the access list, then access is restricted to the s on the list. Do not enter an asterisks (*) to indicate "all s." Doing so causes the Data Domain system to only allow a with that name to be allowed to link to the shared folder. Also, that s and groups must be entered in domain\name format. If there is a space in the name or group, enter the name in quotes. If a group name is specified, the Data Domain system is able to find the in the group by using the "cifs troubleshooting <domain>\\<name>" CLI command. Note the two backslashes used in the "cifs troubleshooting <domain>\\<name>" CLI command. Also, the group is discoverable to the Data Domain system by using the "cifs troubleshooting list-groups" CLI command.



95

Share Access - ing To a is discoverable by the Data Domain system, execute the "cifs troubleshooting " CLI command with the domain and name arguments. Make sure to put in two backslashes as separators between the two elements.



96

Share Access - Testing (Deep Dive) To test access from the Data Domain system, use the "smbclient" bash command. The -U argument is followed by the domain and name. These arguments are then followed by the target server name and share name. Note that the slash is used as a delimiter in this syntax instead of the more familiar backslash. In the example on the screen, we are testing the BKUPSRVR03 from the BACUPNET domain's ability to access the backup share on the localhost, which is the Data Domain system from which this command is being executed. References smbclient man page https://www.samba.org/samba/docs/man/manpages/smbclient.1.html



97

Share Access - ing Share Config When troubleshooting share configuration issues, first check whether the share exists using the cifs share show CLI command. Next, check that the share name points to an existing directory. Review that the client access is correct. Check for any error messages on the Windows client. A system error 53 indicates the client cannot find the share. A system error 59 indicates an unexpected network error occurred. the 's access has not been restricted by Windows Security Settings (ACL).



98

Share Access - ing Share Config (Continued) If the problem still persists, check for authentication issues. Make sure the has privileges to the shared folder. Also make sure the is correct.



99

SMB g - Description SMB g is a feature that allows communications using SMB to be digitally signed at the packet level. Digitally g the packets enables the recipient of the packets to confirm their point of origination and their authenticity. This security mechanism in the SMB protocol helps avoid issues like tampering of packets and "man in the middle" attacks. SMB g States There are currently three states for SMB g - enabled, disabled, and required. The enabled state makes SMB g available if required by the connected device. The required state informs the other connected device that SMB g must be used. The disabled state means that the device will not SMB g, even if the connected device requires it. SMB connections will fail if one device requires SMB g, and the other device has SMB g disabled. Data Domain system for SMB g started with DD OS version 5.2.4. SMB g is uned before DD OS 5.2.4.



100

SMB g - Managing Windows Clients Windows clients use different mechanisms for managing the SMB g feature, depending on the version. Refer to the Microsoft document "Overview of Server Message Block g" for details on a specific version of Windows. Most Windows clients use two registry keys to manage the configuration of the SMB g feature. The registry keys are: EnableSecuritySignature RequireSecuritySignature The location of these keys in the Windows registry is shown on the screen. The settings for these keys determines the SMB configuration state: Enabled •

EnableSecuritySignature = 1

•

RequireSecuritySignature = 0

Required •


•


Disabled •


•




101

SMB g - ing Windows Clients If running a version of DD OS that does not SMB g, the client must not have SMB g configured as required. To determine if SMB g is required by the Windows client, use the Regedit utility to the configuration manually, or use the "reg" Windows command shown on the screen and in the student guide. reg query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanworkstation\parameters On screen is a sample of the output from the "reg query" Windows command: ServiceDll

REG_EXPAND_SZ    %SystemRoot%\System32\wkssvc.dll

ServiceDllUnloadOnStop

REG_DWORD    0x1

EnablePlainText

REG_DWORD    0x0

EnableSecuritySignature

REG_DWORD    0x1

RequireSecuritySignature OtherDomains

REG_DWORD    0x1

REG_MULTI_SZ



102

SMB g - Configuring Windows Clients The registry keys on the Windows client may be configured locally on the Windows client or through centrally managed group policies. You can change the local registry's RequireSecuritySignature setting to zero (0) by using the "Regedit" windows utility, or by using the following "reg" command: reg add HKLM\SYSTEM\CurrentControlSet\Services\lanmanworkstation\parameters /v RequireSecuritySignature /t REG_DWORD /d 0x0 You may need to execute the "Regedit" or "reg" command with the "Run as " option. Refer to the Microsoft document "Overview of Server Message Block g" for details on how to configure SMB g through group policies. Reboot the Windows client after changing this setting.



103

SMB g - Managing the Data Domain system ing the SMB g State on the Data Domain system To the state of SMB g on the Data Domain system, use the "cifs option show" CLI command. The SMB g feature is called "server g" on the Data Domain system. Configuring SMB g State on Data Domain system Use the "cifs option set" CLI command to configure SMB g on the Data Domain system. To disable the SMB g feature, use the "cifs option reset" CLI command. Notes EMC Data Domain encourages that SMB g feature be disabled unless it is required by your enterprise. SMB g may cause a decrease in server performance of around ten percent. The default setting for the "server g" feature is disabled. Because there is a space in the option name, the "server g" option is surrounded by quotes. When configuring the "server g" option, you will not receive an error if you specify an incorrect option value.



104

SMB g - Errors and Messages The Data Domain system clients.log file shows the state of SMB g (server g) when the Windows establishes a connection. [g ‐ server(disabled), client(disabled)] connect to service backup initially as  dd120‐ train1\sys (uid=100, gid=60001800) (pid 24275) If SMB g is required by client but disabled on server, the Windows client displays this error message: System error 1240 has occurred. The  is not authorized to log in from this station. If SMB g is required by server but disabled on client, the Windows client displays this error message: Required by server, disabled on client ‐ System error 5 has occurred.  Access is denied. References Reg Tool on Technet http://technet.microsoft.com/en-us/library/cc732643.aspx 181357 : SMB g on Data Domain https://.emc.com/kb/181357 Overview of Server Message Block g http://.microsoft.com/kb/887429 CIFS http://msdn.microsoft.com/en-us/library/Aa302188.aspx



105

File Access Troubleshooting If the client request is failing with a STATUS_ACCESS_DENIED or STATUS_PRIVIELEGE_NOT_HELD error, check the permissions of the file and the parent directory using the following methods: If, from a Windows client, you are able to gain access to the file or directory as another (such as sys), use Windows Explorer to examine the security properties of the file or parent directory. If necessary, you can check the permissions on the file using the "se dd_xcacls" CLI command. This command allows you to view and modify the object's owner and group SID. It also allows you to view and modify the object's permissions and auditing information from the perspective of the operating system. Check if the file or directory is a replication target using the "replication show config" CLI command. You cannot write to a replication target. that Write Once Read Many (WORM) retention-lock features are not configured on the file. The file cannot be modified or deleted when retention lock has been applied. References 180673 : Mapping a Network Drive to the Data Domain system https://.emc.com/kb/180673 181517 : Unable to Write, Modify, or Delete a Directory or File on a Replication Target https://.emc.com/kb/181517



106

LESSON SUMMARY - Share and File Access Control Troubleshooting This lesson discussed the following topics: • Troubleshooting Share-Level Access Control Issues • Describing and Troubleshooting SMB g • Troubleshooting File-level Access Control Issues



107

Knowledge Check



108

LESSON - CIFS Performance Troubleshooting This lesson discusses the following topics: • Troubleshooting performance degradation due to slow network • Diagnosing session timeout issues • Listing CIFS performance troubleshooting recommendations



109

CIFS Performance - Slow Network When troubleshooting CIFS performance, you need to or eliminate the network and NIC as an issue by using the "net iperf" CLI command. Use the "window-size" option to specify a windows size of 1 megabit. If the test does not yield throughput that is close to the maximum capabilities of the NIC, then check the network for the fault. Another test is to create a file on the Windows client using the "fsutil" Windows program. You can use the command shown on the screen to create a ten megabyte file on the Windows 's desktop. Copy this file to a CIFS shared folder. If you reach the expected write speed, then the issue could lie with the backup application. You may also wish to analyze network traffic using the "net tdump" CLI command. Finally, if the CIFS client is a Linux device, you can run the "lmdd" command. References • fsutil https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/enus/fsutil.mspx?mfr=true • fsutil http://technet.microsoft.com/en-us/library/cc753059.aspx • Wikipedia (iperf) http://en.wikipedia.org/wiki/Iperf • LMDD Man Page http://dev.justmanpage.com/web/man/lmdd.8



110

CIFS Performance - Session Timeout If latency is high in a network, consider changing the CIFS session timeout value on the windows client from the default value of 45 seconds to a value of 3600 second - which is one hour. You can view the SESSTIMEOUT value by executing the "reg query", "regedit", or "regedit32" windows commands. reg query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanworkstation\parameters

If the field is not displayed, it means that the Windows client is using the default value. You can set (or reset) and add the key by executing the "reg add" command as an or by using the "regedit" or "regedit32" program. reg add HKLM\SYSTEM\CurrentControlSet\Services\lanmanworkstation\parameters /v SESSTIMEOUT /t REG_DWORD /d 3600 Even though this step addresses the requirements for most systems, check the Knowledge base article "Setting Network Parameters on the Client for Optimal Throughput with a Data Domain system" for information on most versions of Windows. References 180579 : Setting Network Parameters on the Client for Optimal Throughput with a Data Domain system https://.emc.com/kb/180579 SMB 2.x and SMB 3.0 Timeouts in Windows http://blogs.msdn.com/b/openspecification/archive/2013/03/27/smb-2-x-and-smb-3-0-timeouts-inwindows.aspx



111

CIFS Performance Tuning Consider increasing the T window size for better performance. The knowledgebase article "Network Performance Troubleshooting" provides information on how to configure different operating systems, including Windows, AIX, Red hat, and Solaris. To make the new T window size take effect on the Windows client, restart the machine. Using the "cifs option show" CLI command on the Data Domain system, the maxxmit option is at the maximum value. The maximum value for this option is currently 65536. The output from the "cifs option show" CLI command displays the options that have been changed from their default values at the top. The default values are version dependent. In DD OS 5.1, the maxxmit option has a default value of 16644. In DD OS 5.4, the maxxmit option has a default value of 65536. To change the value of the maxxmit option, use the "cifs option set maxxmit" CLI command. References • 180512 : Network Performance Troubleshooting https://.emc.com/kb/180512



112

LESSON SUMMARY - CIFS Performance Troubleshooting This lesson discussed the following topics: • Troubleshooting performance degradation due to slow network • Diagnosing session timeout issues • Listing CIFS performance troubleshooting recommendations



113

Knowledge Check



114

MODULE SUMMARY - CIFS Troubleshooting You have completed this module. You should now be able to: • Describe CIFS • List possible CIFS problem areas • Troubleshoot CIFS authentication • Troubleshoot CIFS shares • Perform file access troubleshooting



115

MODULE - NFS Troubleshooting Upon completion of this module, you will be able to: •

Describe NFS

•

Configure NFS

•

Troubleshoot NFS



116

LESSON - NFS Overview This lesson discusses the following topics: • Describing NFS and NFSv3 • Listing NFS configuration steps • Describing the nfs add CLI options • Mounting an NFS export on a client • Enabling authentication for NFS



117

NFS Description The NFS network file system protocol was originally developed by Sun Microsystems in 1984. It allows a on a client computer to access files on an NFS server over a network. NFS clients include Unix, Linux, and Windows operating systems. the client is ed by the version of DD OS running on the Data Domain system by reviewing the documentation and release notes. There are four versions of NFS, versions 1, 2, 3, and 4. Data Domain s version 3 (NFSv3). No license is required to use the NFS feature on a Data Domain system. References RFC 1813 - NFS Version 3 Protocol Specification http://tools.ietf.org/html/rfc1813 Wikipedia.org Network File System http://en.wikipedia.org/wiki/Network_File_System



118

What is NFSv3? NFSv3 s 64-bit file sizes and offsets. This enables it to handle files larger than 4 gigabytes in size. NFSv3 also s asynchronous writes on the server. This improves write performance because writes to disk are unacknowledged. The NFS client assumes the write worked unless the NFS server says otherwise. An NFSv3 server returns additional file attributes to the NFS client in response to certain requests. Using this technique, the NFS server helps to avoid the need for the NFS client to issue subsequent requests to obtain the attribute information. NFSv3 servers also read directory plus (READDIRPLUS) operations. This operation causes the NFS server to get file handles and attributes, along with file names, when scanning a directory. References Wikipedia Asynchronous IO http://en.wikipedia.org/wiki/Asynchronous_write



119

Review NFS Status To review the NFS status, follow these steps. First, NFS is enabled by using the nfs status CLI command. This command tells you if NFS is enabled and the number of NFS requests that have been serviced by the system. If necessary, you can enable NFS by using the nfs enable CLI command. The command lets you know if it completed successfully or if NFS is already enabled.



120

Review NFS Exports To review the directories that have been exported and shared with other devices on the network, use the nfs show clients command. This command displays: • A list of exported directories • The clients that are allowed to mount the exported directory • The export options associated with the directory



121

Add NFS Exports Export (share) a directory If you determine that you must export or share a directory on the Data Domain system, you can use the nfs add CLI command. The nfs add CLI command returns the result of the operation and provides error messages as appropriate. The command arguments for the add nfs CLI command are path, client, and options. Path The path argument specifies the location of the target export directory on the Data Domain system. The path starts with /backup, /ddvar, or /data/col1/. Client The client argument identifies the clients that have access to this mount point. Clients may be specified through their IP address, IP address and subnet, host name, domain name, or by the use of the wildcard character which is the asterisk (*). The best practice is to use the host name when specifying clients. Options Options are available to provide more control over the access and management of the exported directory. There is no option keyword associated with the nfs CLI command set. The options are identified by the fact they are in parentheses at the end of the command. Individual items in a list of options must be separated by commas. Spaces are not allowed in the option list. Example (sec=sys,rw,no_root_squash,no_all_squash,secure,nolog)



122

nfs add CLI Command Options When using the "nfs add" CLI command, you may exert more control over the access and management of the exported directory through the use of the net add CLI command options. ro, rw The ro and rw options allow you to grant the assigned clients read-only or read-write access to the exported directory. Unless otherwise specified, the nfs add CLI command enables the rw option. root_squash, no_root_squash The root_squash option squashes root access through the exported directory by mapping the root UID to the anonymous UID. This means that even though the has root access on their local system, they only have the access assigned to the anonymous on the exported directory. Squashing the root UID is done for security purposes. Just because somebody has logged in as root and has root-level access to their local file system does not mean they should have root-level access to the exported directory. The no_root_squash option allows the NFS client's root UID to be used on the exported directory. Unless otherwise specified, the nfs add CLI command enables the no_root_squash option and the root_squash option is disabled. all_squash, no_all_squash The all_squash option causes the system to map a non-root UID from the local NFS client to the anonymous UID on the NFS server when accessing an exported directory. This gives every non-root the same file access privileges. The no_all_squash option allows the non-root UIDs to be used unchanged on the exported directory. Unless otherwise specified, the nfs add CLI command enables the no_all_squash option and the all_squash option is disabled. secure, insecure The secure and insecure options specify the ports from which the NFS requests must originate. When enabled, the secure option only allows access to client requests originating from secure ports - those below 1024. The insecure option allows client requests from any port. Unless otherwise specified, the nfs add CLI command enables the secure option and the insecure option is disabled. Copyright 2015 EMC Corporation. All rights reserved.


123

NFS Add Options (Continued) anonuid, anongid The anonuid and anongid options enable you to map anonymous to a specific UID and GID. sec=sys, sec=krb5, sec=sys:krb5 The sec=sys option directs the system to use local system authentication. The sec=krb5 option directs the system to use Kerberos Version 5 authentication. The sec=sys:krb5 specifies that both security methods are to be used. When both security methods are to be used, to separate the sys and krb5 options with a colon. Unless otherwise specified, the nfs add CLI command enables the sec=sys option and the krb5 option is disabled. log, nolog The log and nolog options enable or disable the NFS log. Unless otherwise specified, the nfs add CLI command enables the nolog option and the log option is disabled.



124

Using the nfs.log Option The nfs.log option was introduced in DD OS version 5.5.1. You can enable NFS logging on a per export basis. This means the activity on some exported directories may be captured while the activity on other exported directories is not. Use the nfs add CLI command with the log option to enable logging. On screen are example CLI commands enabling and disabling the nfs.log option. Log messages are written to the nfs.log file found in the debug directory. You can view the log file with the log view CLI command. Using the nfs.log option makes the system examine all NFS packets on the exported directory. This, of course, can impact the performance of the Data Domain system.



125

nfs add CLI Command - Examples On screen are examples of the nfs add CLI command. Example 1 provides access to all IPv4 and IPv6 clients. Example 2 provides access to all IPv4 and IPv6 clients and uses both local and Kerberos authentication. NFS logging is enabled. Example 3 allows myHost read-only access to the exported directory. myHost can be an IPv4 or IPv6 client. Example 4 allows all workstations in the edu.emc.com domain access to the exported directory. The insecure option means the NFS mount request can originate from any T port. Examples # nfs add /data/col1/nfsGoodTest * All IPv4 and IPv6 clients access # nfs add /data/col1/linuxDir * (sec=sys:krb5,log) All IPv4 and IPv6 clients allowed. Local and kerberos authentication. NFS logging is enabled. # nfs add /data/col1/training myHost (ro) Allow myHost read-only access. myHost can be IPv4 or IPv6. # nfs add /data/col1/linuxDir *.edu.emc.com (insecure) Allow access to all devices in edu.emc.com. The NFS mount request can originate from any T port.



126

nfs add CLI Command Examples (Continued) Example 5 allows the IPv4 device that meets the address and subnet mask requirements to access the export. Example 6 provides access to all IPv6 clients. Example 7 provides access to all IPv6 clients from the specified IPv6 subnet. Example 8 provides access to all IPv6 clients using their link-local address. The prefix, fe80, identifies this as a link-local address. Link-local addresses are not routable, so only clients on the local subnet may access the export. Examples # nfs add /data/col1/test 192.168.1.02/255.255.255.0 •

Allow access to an IPv4 device that meets the address and subnet mask requirements

# nfs add /backup ::/0 •

Allows access to IPv6 clients

# nfs add /backup 2620:0:170:1a01::/64 •

Allows access only from this IPv6 subnet using a 64-bit mask.

# nfs add /backup fe80::/10 •

Allows all IPv6 clients on local IPv6 subnet access through their link-local address (prefix fe80).



127

Mounting an NFS Export on a Linux Client When mounting an NFS export on a Linux client, follow these steps: 1. First, connectivity to the Data Domain system. You can do this using the ping command. 2. Next, decide where you wish the mount-point to be on the Linux client. It can be an existing directory, or a new directory you create. In the example on the screen, a directory named /mtnddsys is created on the Linux client to serve as the mount point. 3. Now, use the Linux mount command to link the exported directory on the Data Domain system with the mount point on the Linux client. The examples on the screen show a mount commands targeting a DNS name, an IPv4 address, and an IPv6 address. 4. The mount command has a number of options that follow the -o flag. All options should be included in one string. Options are separated by commas. No spaces are allowed in the option string. Note: On Sun Solaris systems, specify the lock option instead of the nolock option. Use the nolock option on all other UNIX and Linux systems. Refer to student guide for an explanation of the options shown. Mount command options

hard

This option specifies that the program using the NFS connection should stop and wait for the Data Domain system to come back online, if the Data Domain system is unavailable. The cannot terminate the process waiting for the NFS communication to resume unless the intr option is also specified.

bg

This option causes the device to execute a background mount.

intr

This option allows NFS requests to be interrupted if the server goes down or cannot be reached.

rsize

This option specifies the maximum number of bytes the NFS client can receive when reading data from the Data Domain system.

wsize

This option specifies the maximum number of bytes the NFS client can write to the Data Domain system.

nolock

This option determines that the NFS client cannot lock files to prohibit access by other NFS clients.

proto

This specifies if T or UDP is used for the NFS connection.

vers

This specifies the version of the protocol that will be used.



128

Mounting an NFS Export on a Windows Client NFS is not ed on Windows clients, but there are times when you may need to use a windows client for expediency sake. To access NFS exported directories from a Windows 7 client, use the command shown on the screen. C:> mount ‐o mtype=hard,rsize=32768,wsize=32768,nolock \\ddsys\backup  h: In order to access the NFS server, the Windows client must have the Client for NFS service installed.



129

NFS Authentication Methods ed NFS Authentication Methods There are three authentication methods ed for NFS - Local, Kerberos UNIX, and Kerberos Windows. The Kerberos UNIX authentication method requires a UNIX (Linux) Key Distribution Center (KDC). The Kerberos Windows authentication method requires a Domain Controller (DC). The Data Domain CLI provides you with a number of tools to enable you to configure NFS authentication. CLI command To disable Kerberos authentication, use the authentication kerberos reset CLI command. To configure the Data Domain system for NFS Kerberos UNIX authentication, use the authentication kerberos set realm CLI command with a kdc-type of unix. To configure the Data Domain system for NFS Kerberos Windows authentication, use the authentication kerberos set realm CLI command with a kdc-type of windows.



130

NFS Local Authentication With NFS local authentication, the and 's permissions are defined on the Data Domain system. Local NFS Authentication is enabled by including the sec=sys option in the nfs add CLI command. Configuration Steps To configure NFS Local Authentication, you must first add s to the Data Domain system with the add CLI command. Next, when defining the export, you may include the sec=sys option to indicate the export uses NFS Local authentication. This is the default setting for all exports.



131

NFS Kerberos Windows Authentication When using the Windows Kerberos Authentication method, NFS authentication is performed by the Active Directory (AD) service which usually runs on a Windows Domain Controller (DC). The AD also serves as a Kerberos Distribution Center (KDC). Because of this, the AD, DC, and Windows KDS usually refer to the same function. When the Data Domain system is configured for Kerberos Windows authentication, NFS and CIFS clients and servers use the AD for NFS authentication. Implementation steps The following tasks must be performed in order to successfully implement NFS Kerberos UNIX Authentication. 1. First, create the keytab file on the Windows KDC for the Data Domain system. 2. After creating the Keytab file, copy it to the /ddvar directory on the Data Domain system. The Keytab file must be name krb5.keytab. 3. Next, start the Data Domain system configuration process by resetting the authentication Kerberos configuration to default values by using the authentication kerberos reset CLI command. 4. Now, use the authentication kerberos set realm CLI command to activate Kerberos Windows NFS authentication. Up to three KDCs may be referenced when activating Kerberos Windows NFS Authentication. 5. Next, import the krb5.keytab file by using the authentication kerberos keytab import CLI command. This command moves the krb5.keytab file from the /ddvar directory to the /ddr/etc directory. 6. Now, include the sec=krb5 option when creating an export to invoke Kerberos NFS authentication for clients attempting to access that shared resource. CIFS clients will also use Kerberos authentication. 7. Next, the operation of Kerberos authentication on the Data Domain system. 8. Finally, the krb5.keytab file was deleted from the /ddvar directory.



132

NFS Kerberos Windows Authentication When using the Windows Kerberos Authentication method, NFS authentication is performed by the Active Directory (AD) service which usually runs on a Windows Domain Controller (DC). The AD also serves as a Kerberos Distribution Center (KDC). Because of this, the AD, DC, and Windows KDC usually refer to the same function. When the Data Domain system is configured for Kerberos Windows authentication, NFS and CIFS clients and servers use the AD for NFS authentication. Implementation steps The following tasks must be performed in order to successfully implement NFS Windows Kerberos Authentication. 1. First, create the keytab file on the Windows Active Directory for the Data Domain system. 2. After creating the Keytab file, copy it to the /ddvar directory on the Data Domain system. The Keytab file must be name krb5.keytab. 3. Next, start the Data Domain system configuration process by resetting the authentication Kerberos configuration to default values by using the authentication kerberos reset CLI command. 4. Now, use the authentication kerberos set realm CLI command to activate Kerberos Windows authentication. Up to three KDCs may be referenced. 5. Next, import the krb5.keytab file by using the authentication kerberos keytab import CLI command. This command moves the krb5.keytab file from the /ddvar directory to the /ddr/etc directory. 6. Now, include the sec=krb5 option when creating an export to invoke Kerberos Windows authentication for clients attempting to access that shared resource. CIFS clients will also use Kerberos Windows authentication. 7. Next, the operation of Kerberos authentication on the Data Domain system. 8. Finally, the krb5.keytab file was deleted from the /ddvar directory.



133

NFS Authentication (Continued) A Kerberos keytab file is an encrypted file that is used to enable an NFS server to automatically authenticate with the Key Distribution Center (KDC). Use the following steps create and import a keytab file to the Data Domain system: 1. First, create the krb5.keytab file on the KDC. 2. Next, transfer the keytab file to the /ddvar directory on the Data Domain system. This is a sensitive file, so use a secure transfer method when moving the file into this directory. 3. Now, use the authentication kerberos set realm CLI command to activate Kerberos authentication. The import process If the kdc-type is windows, the krb5.keytab file should be imported automatically. The import process moves the keytab file from the /ddvar directory to the /ddr/etc directory. The /ddr/etc directory is not accessible, providing the file a high degree of security. 4. If the kdc-type is unix, you must manually import the keytab file using the authentication kerberos import keytab CLI command. 5. Test the system to make sure it works. 6. the krb5.keytab file has been deleted from the /ddvar directory. This is a sensitive file and should not be left in a location where it can be compromised.



134

LESSON SUMMARY - NFS Overview This lesson discussed the following topics: • Describing NFS and NFSv3 • Listing NFS configuration steps • Describing the nfs add CLI options • Mounting an NFS export on a client • Enabling authentication for NFS



135

Knowledge Check



136

LESSON - NFS Troubleshooting This lesson discusses the following topics: • Using the showmount command for troubleshooting • Diagnosing NFS Connectivity Issues • Resolving NFS Performance Issues • Describing the effect of the hard and soft mount options



137

Troubleshooting with showmount You can use the showmount command to the NFS configuration of the Data Domain system. The showmount command provides a list of the remote directories that have been mounted (attached) to the target system (local or remote) or the list of directories that have been exported (shared) by the target. The showmount command also lists the clients allowed to mount the exported directory. The showmount command is available on many clients including Linux and Windows. On windows clients, it may require the activation of the "Services for NFS" feature. This can be activated by navigating to the "Control -> Programs and Features -> Turn Windows Features on or off" and enabling this functionality. Export added when directory doesn't exist The showmount command can be useful if there is a discrepancy between what the Data Domain system is reporting and what the client is seeing. For example , if an export is added, but the directory doesn't exist, the "nfs add" CLI command informs the sys of the problem. If the sys does not see this warning, they will be unaware of the issue. CLI fails to show nfsBadTest points to non-existent path A subsequent use of the "nfs show client" CLI command lists the export, but fails to show that there is a problem with the path. Note: The browser-based GUI does show that there is a problem with the path. Client showmount command cannot see nfsBadTest If you interrogate the Data Domain system from a client using the showmount command, you will see that the client does not see the export with the bad path. As you can see, you should test the validity of the exports from a client to ensure the client can see what the Data Domain system is configured to . References • showmount man pages http://www.unix.com/man-page/freebsd/8/showmount/ • Utilities and SDK for Subsystem for UNIX-based Applications in Microsoft Windows 7 and Windows Server 2008 R2 http://www.microsoft.com/en-us//details.aspx?id=2391



138

Troubleshooting NFS Connectivity connectivity with ping When troubleshooting NFS connectivity, you can use the ping and rpcinfo commands. The ping command can be used to general connectivity. Gather NFS rpc program info from DDSYS Next, use the rpcinfo command on the NFS client to gather information about the NFS program on the Data Domain system. The syntax for the rpcinfo command is shown on the screen. The -t flag instructs the rpcinfo program to use T. The -u flag instructs the rpcinfo command to use UDP as the transport protocol. Use both -t and -u flags to both transport protocols are functional. The target_host field is replaced by the IP address or DNS name of the target Data Domain system. Finally, the rpcinfo command identifies the target service. In the first example on the screen, the NFS service is targeted. In the second example, the mountd service is targeted. Gather mountd rpc program info from DDSYS To gather mountd rpc program information from the Data Domain system, use the rpcinfo command targeting the mountd program. Results from rpcinfo command The rpcinfo command returns information and status messages about the target program. The display shows what is returned when NFS and mountd are running correctly. The results of the rpcinfo command also include the rpc program number which, for NFS, is 100003 and for the mountd program is 100005. References rpc.mountd man page - http://linux.die.net/man/8/rpc.mountd rpcinfo man page - http://linux.die.net/man/8/rpcinfo



139

Troubleshooting rpcinfo Errors Error - rpcinfo: RPC: Program not ed If the rpcinfo command returns an error that states that the program is not ed, it may be that the Data Domain filesystem is down. Error: RPC: Port mapper failure - RPC: Timed out. An rpcinfo error that identifies a port mapper failure could indicate the NFS service is not enabled on the Data Domain system.



140

Running rpcinfo in BASH If necessary, the rpcinfo command can be run on the Data Domain system in BASH. DD OS 5.5.1 introduces for rpcinfo -s Starting with DD OS 5.5.1, the -s option is ed. !# /usr/bin/rpcinfo -s The -s option causes the rpcinfo program to list the RPC program number, version number, ed network protocols (or netid), service name, and the owner of the RPC programs running on the Data Domain system. The IPv6 netids end with the number 6. The IPv4 netids do not.



141

NFS Performance Issues NFS performance problems are most often due to the network or client-side mount options. Network issues To diagnose network issues, you can use the net show stats, nfs show stats, sys show stats view net, and sys show stats view nfs CLI commands. You can also use network analysis tools such as tdump, Wireshark, and iPerf. mount sync / async option Client-side mount options can have a significant effect on performance. For example, the mount command s a sync option. This option causes all I/O to the Data Domain system to be be done synchronously. This means every I/O has to be acknowledge by the Data Domain system before the NFS client will send another request. If the mount command contains the sync option, consider using the async option instead. The async option causes all I/O to the Data Domain system to be done asynchronously; that is that I/O request do not require acknowledgment before the next request is made. Reference Mount command - http://ss64.com/bash/mount.html



142

Hard / Soft Mount Option Issues Symptom If performance symptoms include commands like df hanging on the client or the Data Domain system does not respond to ping, then it may be due to the use of the mount hard option. Hard Option Effects The hard option causes the NFS client to retry requests indefinitely. This means that any command trying to access the exported directory on the Data Domain system will seem to hang as it waits for its request to complete.



143

Hard / Soft Mount Option Issue Resolution Use soft option The resolution to this problem could be to change the hard option used in the mount command to the soft option. Effects of the soft option The soft option has the following effects: First, it causes the mount command to fail after the configured number of retransmission requests have been sent. It can also cause silent data corruption in certain cases. Use when client responsiveness highest priority Because of these possible issues, use the soft option only when client responsiveness is higher priority than data integrity. Mitigating soft option risks You can mitigate the risks posed by using the soft option by specifying the mount use NFS over T. Use the t mount option if it is ed by your NFS client. If you choose to still use UDP as the transfer protocol, you can increase the number of times the NFS client will retransmit a request using the mount command's retrans option. If the retrans option is ed on your NFS client, try a value of 6 or greater. The default retrans value is 3. Reference nfs man page - http://linux.die.net/man/5/nfs



144

LESSON SUMMARY - NFS Troubleshooting This lesson discussed the following topics: • Using the showmount command for troubleshooting • Diagnosing NFS Connectivity Issues • Resolving NFS Performance Issues • Describing the effect of the hard and soft mount options



145

Knowledge Check



146

MODULE SUMMARY - NFS Troubleshooting You have completed this module. You should now be able to: •

Describe NFS

•

Configure NFS

•

Troubleshoot NFS



147

Course Summary You have completed this course. You should now be able to: Perform CIFS Troubleshooting Perform NFS Troubleshooting



148

Knowledge Check



149

This concludes the Training. Thank you for your participation.



150

Data-domain-cifs-and-nfs-troubleshooting-sg.pdf 601h4k

Overview 3e4r5l

More details w3441

More Documents from "Abhishek Pubbisetty" 6e1442

General Data Domain Operating System Upgrade Instructions 5o4y6s

Ddhardware_maintaince_studentguide 323z20

Data-domain-cifs-and-nfs-troubleshooting-sg.pdf 601h4k

Lecture 1 - The Concept Of Strategy 5x6h3c

Cleartrip Train E-ticket 25te

Boot Animation 2m6p6e